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abstract 


An experiment to study how automation, when used in conjunction with datalink 
for the delivery of ATC clearance amendments, affects the situational awareness of 
aircrews was conducted. The study was focused on the relationship of situational 
awareness to automated Flight Management System (FMS) programming and the readback 
of ATC clearances. Situational awareness was tested by issuing nominally unacceptable 
ATC clearances and measuring whether the error was detected by the subject pilots. The 
experiment also varied the mode of clearance delivery: Verbal, Textual, and Graphical. 

The error detection performance and pilot preference results indicate that the automated 
programming of the FMS may be superior to manual programming. It is believed that 
automated FMS programming may relieve some of the cognitive load, allowing pilots to 
concentrate on the strategic implications of a clearance amendment. Also, readback appears 
to have value, but the small sample size precludes a definite conclusion. Furthermore, 
because textual and graphical modes of delivery offer different but complementary 
advantages for cognitive processing, a combination of these modes of delivery may be 
advantageous in a datalink presentation. 


This document is based on the thesis of Edward C. Hahn submitted in partial 
fulfillment of the degree of Master of Science in Aeronautics and Astronautics at the 
Massachusetts Institute of Technology. 
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1 INTRODUCTION 


The Federal Aviation Administration (FAA) has mandated the use of digital ground- 
to-air datalink for Air Traffic Control (ATC) services in the mid- 1990’s timeframe. 

Possible benefits of datalink include reducing voice congestion and information transfer 
errors associated with VHF radio communications. In addition, studies have shown that 
datalink has other potential benefits, such as providing an independent reference of the 
aircraft clearance [1,2]. However, there is some concern that datalink, especially when 
combined with automation, may actually decrease the crew’s level of situational awareness 

[3]. 

The Aeronautical Systems Laboratory of the Massachusetts Institute of Technology 
became involved in the investigation of the possible benefits of datalink delivery of ATC 
clearance amendments in 1989 [4, 5]. In an initial study, datalink combined with 
automation was found to provide a significant decrease in the time needed by the subject 
pilots to process ATC clearances. However, one of the side effects of the most automated 
case was a possible degradation of the subjects’ situational awareness as indicated by the 
ability to detect nominally unacceptable clearances. This thesis document describes the 
further investigation of the effects of automation on situational awareness when combined 
with datalink delivery of ATC clearance amendments. 

Chapter 2 provides background information about datalink transmission of ATC 
messages and automation. Chapter 3 explains the factors manipulated and data measured in 
the experiment. Chapter 4 provides information about the test facilities, procedures, and 
scenarios. Chapter 5 discusses the performance results of the experiment, while Chapter 6 
details the subjective information provided by the subject pilots. Chapter 7 identifies other 
research issues prompted by pilot comments during the execution of experiment. Chapter 8 
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summarizes the findings of the experiments and recommends further activities in the 
datalink transmission of ATC clearances. 
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2 BACKGROUND 


2.1 Datalink ATC Clearances & Situational awareness 

Datalink communication of ATC clearances is being developed by the FAA to 
alleviate voice congestion of VHF communication frequencies, as well as to reduce 
potential transmission errors in the delivery of clearance amendment information to aircraft. 
Voice congestion would be reduced through datalink s selective addressing feature, which 
imparts to a particular aircraft only information which is specifically intended for that 
aircraft. Additionally, because the datalink system could transmit clearance information in a 
digital form, potential sources of confusion in ATC-to-aircraft communications, such as 
transcription errors, could be eliminated. 

Past studies have shown that pilots are receptive to the judicious use of datalink for 
the delivery of ATC messages [1, 2, 4, 5, 6]. Waller and Lohr conducted an simulation 
study in 1989 using datalink transmission of ATC messages which concluded, “both the 
pilot and copilot favored ... datalink operations for routine ATC message exchange. In 
addition, experienced pilots found that the use of datalink decreased workload [2]. Other 
studies have found that a substantial reduction in operational errors was potentially 
achievable with datalink [1,6]. 

Along with reducing voice traffic and communications errors, another possible 
benefit of datalink transmission of ATC clearances is that, because digital information is 
easily stored and recalled, an independent record of aircraft clearance amendments could be 
implemented in the flight deck. Anecdotes about aircrews occasionally “mishearing” 
clearance amendment information are not uncommon [7, 10]. A record keeping capability, 
which datalink avionics could easily provide, would reduce potential safety hazards 
resulting from erroneous clearance interpretation. 


10 


Furthermore, the FAA and the National Aeronautics and Space Administration 
(NASA) are studying systems which would automatically gate clearance amendment 
information into the onboard Flight Management System (FMS). Knox and Scanlon 
conducted a series of flight tests at NASA/Langley Research Center in 1990 designed to 
validate the concept of automated loading of clearance data into the aircraft FMS. Among 
the possible benefits cited with automated datalink was that, “the capability of transferring 
ATC tactical and strategic information into the FMS ... with a single button push, at the 
pilot’s discretion, was a significant work saver.” [1] 


While all proposed datalink systems would require pilot authorization before an 
aircraft would automatically execute a new clearance, there is some concern that pilots 
could become less involved in the clearance amendment processing loop and therefore may 
not be fully aware of the consequences of new amendments. Figure 2.1 shows the ATC- 
to-aircraft communications loop, which currently requires all clearance information to be 
processed by the crew. However, automation of datalink may inadvertently exclude the 
crew from the loop because they would assume a supervisory rather than participatory role 
in clearance communication. 
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FIGURE 2.1. THE ATC-TO-AIRCRAFT COMMUNICATIONS LOOP 
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2.2 Prior Experiments OnDatalink Situational awareness 

Prior simulation studies and flight tests have marnTy concentrated on differences in 
pilot response times and message transaction frequency between voice and datalink. 
Unfortunately, there have been few studies concentrating on crew situational awareness in 
the datalink environment, and the information on situational awareness in datalink 
experiments has been largely anecdotal [1,2, 6]. 

An experiment which included detection of flawed ATC clearances as a measure of 
situational awareness was performed by Chandra [4, 5], and was used as a foundation for 
this study. This experiment, in which six transport pilots participated, focused on the 
effect of automation on the time required by the subject pilot to process clearance 
amendments, with workload and situational awareness as secondary measurements. Three 
levels of automation were used in conjunction with three clearance delivery presentation 
modes as the independent variables. These were: verbal (voice), textual (alphanumerics), 
and graphical (pictorial). Each delivery mode had a distinct, fixed procedure associated 
with it. 

A total of 60 erroneous clearances were issued by ATC. The data indicated that, 
while automation and lack of a readback significantly reduced processing time, these 
factors may have been detrimental to error detection performance (Figure 2.2). However, 
the experiment was unable to substantiate any trends between automation, readback, and 
situational awareness because the number of tests was insufficient for statistical 
significance. In addition, because the factors of automation level and presentation mode 
were not independently varied, the results were confounded. 
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Graphic/No Readback, Automation 


Text/No Readback, No Automation 


Verbal/Readback, No Automation 



% of Cases 


FIGURE 2.2. UNDETECTED CLEARANCE ERRORS - RESULTS FROM A PRIOR SIMULATION 
STUDY CONDUCTED BY CHANDRA (REFORMATTED FROM [5], WITH PERMISSION OF THE 
AUTHOR) 
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3 EXPERIMENTAL METHOD 
3.1 Overview 

Because of the possible conflict between automation and situational awareness in 
the Chandra experiment, a more direct investigation of the effect of automation on the 
datalink delivery of ATC clearances was undertaken. A simulation study using active 
airline pilots as subjects was designed. The experiment measured situational awareness as 
indicated by the ability of the pilot to detect erroneous ATC clearances. The study varied 
procedural elements, modes of delivery, and type of error. 

The general testing protocol was: the subject was presented with nominally 
unacceptable ATC clearances intermixed with a series of acceptable clearance amendments 
while in the descent phase of operations. The ability of the pilot to recognize the errors was 
recorded as the dependent variable 1 . Additionally, subjective ratings and comments by the 
subjects were collected. The independent variables in the experiment were chosen to 
modify the number and type of mental review of the clearance. This was accomplished by 
specifying whether or not the pilot needed to program the clearance into the FMS and 
whether or not the clearance procedure required a readback to ATC. Furthermore, the 
mode of clearance delivery to the pilot was varied among verbal, textual, and graphical. 

3.2 Dependent Measures 

The primary measure of situational awareness was the subject’s ability to detect the 
erroneous clearances. The baseline clearance amendment procedure was modeled as three 
steps: 1) the pilot receives the amendment. 2) The pilot reads the information back to 

l The testing protocol incorporated the original idea that the pilot s situational awareness was proportional 
to the number of cognitive reviews of the information that he accomplishes in the course of processing an 
amendment. Initially, there was an expectation that the number of reviews was more important than type 
of review. However, in the course of the experiment it became apparent that type of review was more 
important than the simple number of reviews. 
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ATC. 3) The pilot programs the FMS. Some errors were detected immediately, others 
were detected only after some additional review, and the remainder were never detected. 
Thus, three scoring divisions were used: initially, finally, and never detected. The 
aggregate percentages of error detection performance in each division were used as the 
figures-of-merit to describe the overall level of situational awareness. 

Each nominally unacceptable clearance amendment, or “error”, was scored as an 
initial detection if the subject rejected the clearance before either performing a readback or 
programming the FMS (i.e. immediately upon the initial review of the information). If the 
subject initially accepted the clearance, read the clearance back to ATC, and/or 
accomplished FMS programming, but later rejected the clearance amendment, the error was 
scored as a final detection. Lastly, if the pilot never indicated awareness that the clearance 
was unacceptable, the error was scored as never detected. It should be noted that scenario 
continuity required the delivery of a new clearance after a predetermined time interval. 

Thus, the performance may be uniformly biased toward never detected. 

In addition to detection performance of the unacceptable clearance amendments, 
pilots were asked during the experiment to give subjective evaluation scores to the various 
procedures and presentation modes. Between each scenario, subjects were given the 
opportunity to rate each combination of delivery mode and procedure in terms of overall 
effectiveness, time efficiency, and situational awareness. They were also asked to 
comment about aspects which they desired or disliked about the delivery mode, 
combination of readback and method of FMS programming, and scenario. The ratings 
were based on a seven point scale, with a “1” rating signifying an “unsafe” delivery mode 
and/or procedure. A “4” denoted a rating comparable to current ATC clearance amendment 
procedures, and a “7” rating meant the pilot thought the mode and procedure were an 
optimum combination. 
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At the conclusion of the experiment, subjects were also asked to both rank order 
and assign a numerical rating to the procedures and delivery modes. The subjects were 
asked to rate each from 1 (lowest) to 10 (highest) for each. In addition, subjects were 
asked to make general comments about the simulation and bring up issues which they felt 
were important to datalink. 
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3.3 Independent Variables 


3.3.1 Amendment Procedure 


The procedural requirements for readback and automated FMS programming were 
changed between scenarios to yield differing types and numbers of review. This yielded 
four procedures with varying levels of review, as summarized in Table 3.1. It should be 
noted that procedure number 4 is essentially the current procedure used by air crews in 
normal operations, and procedure 3 (no readback / manual programming) was only tested 
for the verbal mode because of experimental time constraints. 


TABLE 3.1: SUMMARY OF PROCEDURES USED IN THE EXPERIMENT 

Name 

Readback? 

Manual FMS? 


# of Reviews 

Procedure 1 

No 

No 

No Readback / Automated 
FMS Programming 

1 

Procedure 2 

Yes 

No 

Readback / Automated FMS 
Programming 

2 

Procedure 3 

No 

Yes 

No Readback / Manual FMS 
Programming 

2 

MH 

Yes 

Yes 

Readback / Manual FMS 
Programming 

3 


Manual FMS programming necessitated detailed interaction with the clearance 
amendment information, as it required the pilot to type all of the specific clearance 
information elements (such as intersection names or crossing restriction altitudes) into the 
aircraft FMS via the keyboard. In contrast, readback was a simpler procedural 
requirement: pilots simply repeated the clearance back to ATC by voice. Thus, manual 
FMS programming and readback were expected to require different types of mental 
processing on the part of the subject. In this way, the effect of different kinds of cognitive 
review were included in the test plan. 
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3.3.2 MODE OF DELIVERY 


The other independent variable was delivery mode, or how the information was 
displayed to the pilot. The modes evaluated included verbal, textual, and graphical. 

The verbal mode was used as a baseline and was identical to the current VHF radio 
procedure. However, the simulation did not include any message transactions between 
other aircraft on the frequency and ATC. All controller messages conformed to the current 
ATC lexicon. An example of a verbal clearance is shown in Figure 3.1. 


(SPOKEN) "Direct Hartford VOR, direct Boston VOR; after 
Boston expect vectors to ILS Runway two-seven; cross 
Boston at and maintain seven thousand and two-hundred te 
knots" 

FIGURE 3.1. EXAMPLE OF VERBAL CLEARANCE DELIVERY 


Textual clearances were shown on a separate dedicated display. The messages 
were exact textual transcriptions of the corresponding verbal clearances, with additional 
text-specific features (these are detailed in Section 4.2). An example of a textual clearance 
is shown in Figure 3.2. 

direct HARTFORD (HFD), direct BOSTON (BOS); after BOSTON 
expect vectors to ILS Runway 27; cross BOSTON at and maintain 
7000 and 210 knots. 

FIGURE 3.2. EXAMPLE OF TEXTUAL CLEARANCE DELIVERY 


The graphical mode depicted the assigned routing on the aircraft Electronic 
Horizontal Situation Indicator (EHSI). In addition, altitude and speed commands were 
displayed on the altimeter and airspeed indicator, respectively. Since graphical delivery 
distributed clearance information to several different flight displays, special care was taken 
to make the amendment appear distinct from the existing symbology. All clearance 
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amendment information was displayed in an alternating green/orange color at a rate of 1 
Hz. (Additional graphical clearance implementation information is discussed in Section 
4.2.) An example is shown in Figure 3.3. 



FIGURE 3.3. EXAMPLE OF GRAPHICAL CLEARANCE DELIVERY 


3.3.3 TEST MATRIX 

The resulting test matrix combined the four different procedures with the three 
modes of delivery (Table 3.2). 
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4 Experimental protocol 

4.1 Facilities [8] 


The experiment was conducted using the MTT Aeronautical Systems Laboratory 
(ASL) Advanced Cockpit Simulator (Figure 4.1). The simulator facility was used to 
provide pilots with an environment consistent with flight operations in modem transport 
aircraft. 
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FIGURE 4. 1 . THE MIT ADVANCED COCKPIT SIMULATOR 
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The MIT facility is a part-task simulator based on Boeing 757 / 767 and 747-400 
flight displays. The facility utilizes three computers and several control panels to emulate 
the autoflight systems, and was developed over a 3 year period by a number of graduate 
and undergraduate students. 

A Silicon Graphics Personal IRIS 4D/25GT was used to simulate the aircraft 
dynamics and present the primary flight displays. Airspeed, altitude, and vertical speed 
were indicated using tape displays similar to those found on the 747-400. An Attitude 
Director Indicator (ADI) was provided, and was used to display the artificial horizon, 
ground speed, radio altitude, and Instrument Landing System (ILS) localizer and glideslope 
deviations. 

As shown in Figure 4.1, the Electronic Horizontal Situation Indicator (EHSI) was 
located below the ADI, as in the 757 or 767. The EHSI displayed the 757 / 767 map 
mode, including aircraft heading, ground track, programmed route, and weather radar 
reflectivity (WXR). 

The simulator included a Flight Management System which was interfaced through 
an IBM PC-XT Control Display Unit (CDU). This device replicated the major path 
management functions available on the Boeing 757 / 767. This included adding, deleting, 
and modifying waypoints, setting crossing restrictions, and changing destinations and 
runways. As part of the datalink functionality, automated clearance amendment route 
loading was provided in the relevant scenarios. This allowed the subject to approve and 
execute a clearance amendment with two keystrokes. It should be understood that because 
the keyboard layout on the IBM is different from the layout on actual CDUs, there is 
probably some bias in the performance results for the manual programming cases. 

Additional flight control inputs could be made using an emulation of the Boeing 757 
/ 767 Mode Control Panel (MCP). Modes were available to the pilot to command airspeed, 
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altitude, heading, and vertical speed. The aircraft autopilot modes could be selected as 
well, including LNAV (automatic lateral flight path navigation) and VNAV (automatic 
vertical flight path navigation), altitude capture and hold, vertical speed, heading select and 
hold, and localizer and glideslope intercept. 

An IRIS 2400T was used as a simple datalink Air Traffic Control workstation. Its 
primary functions were to allow a remote researcher to monitor the flight progress made by 
the experimental subject, and to transmit datalink messages to the aircraft simulator. A 
mouse-based graphical user interface provided the ability to select and deselect navigational 
information, to determine the aircraft location relative to a scenario reference point, and, as 
illustrated in Figure 4.2, to select and specify content and format of the scripted datalink 
messages. It should be noted that this display was not intended to reproduce any actual or 
proposed advanced ATC workstation. 


Display Control 
Buttons - allows 
information on the ATC 
display to be selected 
and deselected 




NAME 

ROSE 

APT 

NAV 

WPT 

WXR 

AMEND 


VERBAL 


TEXT 

(#) graph 


Delivery Mode and 
FMS Programming 
Selectors - sets 
delivery mode and 
method of FMS 
programming 


I AUTO 
AMEND 




1 

ARM 

SEND 


Clearance Amendment 
Selectors - selects the 
clearance to be 
delivered from a list of 
scripted clearances 


Datalink Clearance 
Delivery Command 
Buttons - clearance 
amendment is first 
"armed", then sent 
with these buttons 


FIGURE 4.2. THE DATALINK ATC WORKSTATION CONTROL PANEL 
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Global scenario parameters such as mode of delivery and automated FMS 
programming were generally set by the controller at the beginning of a scenario. During a 
typical datalink clearance amendment sequence, the controller monitored the subject aircraft 
until it reached a location predetermined by the scenario script. He then selected the 
appropriate preformatted scripted clearance using the “amendment selection buttons. The 
selected clearance amendment appeared on the map display for verification with the 
scenario script. The controller then armed the datalink system by activating the “arm” 
button, and sent the amendment via the “send” button. The controller then monitored the 
voice channel for a readback (if required by the scenario). If the scripted clearance 
amendment was designed to be nominally unacceptable, the controller could choose the 
appropriate ATC correcting action by sending a new clearance, thus repeating the above 
steps. 

4.2 Datalink Clearance Delivery Implementation 

Color reproductions of the figures in this section can be found in Appendix E. 

4.2.1 Textual delivery Mode implementation 

The dedicated textual display exhibited the clearance information using a white, 14- 
point Helvetica Bold typeface on a black background for maximum contrast. The format 
used in the implementation of textual clearance delivery was a transcription of the current 
ATC voice lexicon, with some additional enhancements. 

All numerical data was written in digital, rather than textual, form (eg. “3500”, not 
“three-thousand, five hundred”). Navigational aids and intersections were emphasized by 
presentation in capital letters to distinguish them from procedural phraseology. 
Furthermore, a navigational aid which was part of the clearance routing included its 
identifier placed in parenthesis after the identifier name. An example of a textual clearance 
amendment is shown in Figure 4.3. 
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direct HARTFORD (HFD), direct BOSTON (BOS); after BOSTON 
expect vectors to ILS Runway 27; cross BOSTON at and maintain 
. 7000 and 210 knots. 

FIGURE 4.3. EXAMPLE OF TEXTUAL CLEARANCE DELIVERY 

4.2.2 GRAPHICAL DELIVERY MODE IMPLEMENTATION 

In the implementation of graphical clearance delivery, special care was taken to 
display the clearance amendment in a distinct format. All clearance amendment 
information, for example, was displayed in a green and orange color alternating at a rate of 
1 Hz. While routing information of the EHSI in the clearance amendment used existing 
symbol shapes for waypoints and active route presentation, some information elements 
present in ATC clearances required the creation of new symbols. Crossing restrictions or 
runway changes appeared as superscripts to the associated waypoint or airport in the same 
format as the FMS display (Figure 4.4). Climb and descent commands or airspeed 
changes were shown on the altitude or airspeed tape displays with arrow symbols and the 
assigned altitude or airspeed (in the alternating green and orange color). An example of a 
descent to 14000 feet is shown in Figure 4.5. Heading vectors were given as arrows 
pointing horizontally in the direction of the turn, with the assigned heading displayed at the 
tip of the arrow. Simultaneously, a flashing heading “bug” also appeared on the Electronic 
Horizontal Situation Indicator (EHSI) highlighting the assigned heading (Figure 4.6). 
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FIGURE 4.4. EXAMPLE OF A GRAPHICAL CROSSING RESTRICTION AND RUNWAY CHANGE 




FIGURE 4.5. EXAMPLE OF A GRAPHICAL DESCENT TO 14000 FEET 
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FIGURE 4.6. EXAMPLE OF A GRAPHICAL HEADING VECTOR 

4.2.3 Display behavior during A Clearance amendment 

In a typical clearance amendment sequence with graphical or textual delivery, the 
displays behaved in the following manner. When the clearance amendment was initially 
transmitted to the aircraft, a annunciation light turned on in the alternating green/orange 
color with the simultaneous playback of a digitized voice alert (“A-T-C Message”)- On the 
simulator displays, the clearance amendment appeared with the appropriate text or 
symbology. The subject then read the clearance back to ATC if the procedure required a 
readback, and then pressed an “accept” or “reject” key to let ATC know his intentions. If 
the reject key was pressed, the clearance amendment annunciation light was extinguished 
and the displays were cleared of all clearance amendment information. However, if the 
subject accepted the amendment, the clearance text or symbology would remain lit until he 
specifically turned it off with a press of the “reset amendment” key. A second press of this 
key allowed the subject to recall the most recent clearance amendment as a record of the 
aircraft’s present clearance. 
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4.3 Experimental Procedure 


The experiment was a “within subject” design. Each subject flew the 10 scenarios 
required to fill the entire test matrix to control for differences between subjects. To ensure 
uniform notification, each amendment was annunciated using aural and visual alerts 
regardless of delivery mode or procedure. Each experimental run began during descent, 
approximately 1 20 nautical miles from the destination airport (thus requiring approximately 
twenty minutes to complete). The subject was provided with the appropriate charts and 
approach plates for the scenario destination. After each scenario, subjects were asked for 
comments on the preceding scenario. Prior to the following run, the subject was briefed on 
the next scenario (eg. destination, runway, ATIS (Automated Terminal Information 
Service)) and on the procedure which was to be used for the particular experimental run. A 
placard summarizing the procedure was put next to the simulator displays for reference. 
During the experiment, one researcher acted as the Pilot-Not-Flying (PNF), and another 
researcher acted as a controller at the datalink ATC control station. 

4.4 Scenarios 

The scenarios involved flights within the Northeast Corridor of the United States 
(i.e. the airspace between Washington DC, New York City, and Boston), with heavy 
traffic and weather in the entire region. Each scenario included a total of five clearance 
amendments, of which two were nominally unacceptable (i.e. an error ). One error in 
each experimental run involved a clearance into weather, while the other was related to 
routing. During unacceptable routing clearances, pilots were given one of the following 
types of errors: 1) clearance to an incorrect initial fix to an approach for landing, 2) 
clearance to an incorrect destination, or 3) an illogical routing which headed the aircraft in a 
direction opposite to the intended flight path. 
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Twelve simulator scenarios were developed, including four scenarios with each 
type of routing error. In this way, the order of scenario presentation between subjects 
could be altered to control for learning effects. Care was taken in the design so that 
scenarios with the same type of routing error had equivalent difficulty while simultaneously 
ensuring that it maintained fidelity with the current ATC environment Scenarios which 
included incorrect approach fix errors were classified as Type A scenarios. Type B 
scenarios contained an illogical routing error. Incorrect destination errors defined Type C 
scenarios, and all twelve scenarios are summarized in Table 4.1. 


TABLE 4.1: SCENARIO SUMMARY 


e mmsssssam 

Destination 

Code (Appendix A&C) 

Incorrect Approach Fix 

Boston / Logan Int’l 

Boston A 


Incorrect Approach Fix 

Baltimore-Washington Int’l 

BWI A 

(a.bwi) 

Incorrect Approach Fix 

New York / LaGuardia 

LaGuardia A 

(alga) 

Incorrect Approach Fix 

New York / Kennedy Int’l 

Kennedy A 

(a.jfk) 


Boston / Logan Int’l 

Boston B 

(b.bos) 


Washington National 

National B 

(b.dca) 

WBBBBSKKM 

Newark Int’l 

Newark B 

(b.ewr) 


Philadelphia Int’l 

Philadelphia B 

(b.phl) 

Destination Error 

Hartford / Bradley Int’l 

Bradley C 

(c.bdl) 

Destination Error 

Washington National 

National C 

(c.dca) 

Destination Error 

New York / LaGuardia 

LaGuardia C 

(c.lga) 

Destination Error 

Philadelphia Int’l 

Philadelphia C 

(c.phl) 


IH 



































4.4.1 Weather Errors 


In weather errors, pilots were routed into regions of precipitation, indicated as 
“yellow” or “red” reflectivity levels on the EHSI weather radar display. Flight crews 
normally attempt to avoid high reflectivity areas because of turbulence, rain, lightning, and 
icing. In the example scenario shown in Figure 4.7, an aircraft was initially proceeding, 

“direct Providence VOR, direct Boston VOR, expect vectors for ILS runway 

4 r” at Boston / Logan International Airport. ATC rerouted the aircraft: “direct drunk 

intersection, direct TONNI intersection, expect vectors to ILS Runway 

27”. This clearance would normally be considered unacceptable because the amendment 
routes the aircraft through a line of thunderstorms. 


Boston / Logan 
International Airport 


Providence VOR 



Subject 

Aircral 


TONNI 

Intersection 


DRUNK 

Intersection 


Amended 
Flight Path 


Line of 

Thunderstorms 


FIGURE 4.7. EXAMPLE CLEARANCE ROUTING INTO WEATHER 
NOT TO SCALE 
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4.4.2 Incorrect Approach Fix Errors 

Inappropriate approach fix errors were characterized by the aircraft being cleared to 
an initial approach fix for one approach while simultaneously being cleared by ATC for a 
different approach. Thus, a discontinuity was created in the flight path between the last 
waypoint and the specified approach path. For example, in Figure 4.8, an aircraft was 

proceeding: “direct RUETT intersection, expect vectors for the ILS Runway 
33L approach” at Baltimore-Washington International Airport ATC then cleared the 
aircraft: “direct BALTO intersection, ... cleared for the ILS Runway 33 
Left approach”. Since BALTO intersection is on the approach for runway 28 (and not 
runway 33L, where the corresponding fix is RUETT), a discontinuity in the path is 
created, and therefore the clearance is unacceptable. 


Baltimore-Washington 
International Airport 


BALTO 
Approach Fix 
(Cannot Execute 
ILS 33L Approach 
from here) 



Subject Aircraft 


FIGURE 4.8. EXAMPLE INCORRECT APPROACH FIX ERROR 
NOT TO SCALE 
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4.4.3 INCORRECT DESTINATION ERRORS 


Erroneous clearances to an incorrect destination were always given as an expect 
vectors” clearance to a runway number which did not exist at the intended destination, but 
which did exist at a nearby airport. For example in Figure 4.9, a pilot was enroute to 

Washington National Airport via: “direct Lancaster VOR, expect vectors for 
River Visual Runway is approach”. The flight was then instructed (erroneously) to 
proceed to Harrisburg, Pennsylvania by the following clearance: “direct Lancaster 

VOR, direct BAARN intersection; after BAARN expect vectors to ILS runway 

13”. Since Washington National has no runway 13, there is an unambiguous error in the 
clearance. 


Subject Aircraft 


Harrisburg, PA/ 
Middletown Airport 
RUNWAY 13 


Amended 
Flight Path 



Washington / National Airport 
(RUNWAY 13 non-existent) 


FIGURE 4.9. EXAMPLE INCORRECT DESTINATION ERROR 
NOT TO SCALE 
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4.4.4 ILLOGICAL ROUTING ERRORS 


Illogical routings were characterized by the aircraft being cleared to a point 
inconsistent with the direction to the airport and the current route of flight For example, in 
Figure 4.10, a subject was enroute to Boston via: “direct Putnam vor, direct 
Boston vor”. ATC then instructed him to proceed “direct Hartford VOR, direct 
Boston VOR; after Boston expect vectors to ILS runway 27 ...” when aircraft 
was approaching the Putnam VOR. In the course of the experiment it became clear that 
this kind of routing error is ambiguous. Whereas the other types of errors were clearly 
erroneous, pilots are often given apparently illogical vectors for sequencing and spacing in 
busy terminal areas. One pilot even stated that when flying into the New York Terminal 
Control Area, aircrews “expect anything” because of the complexity of the airspace. 
Because of this ambiguity, the data for the illogical routing errors was not included in the 
routing error analysis. 

Boston / Logan 



FIGURE 4.10. EXAMPLE ILLOGICAL ROUTING ERROR 
NOT TO SCALE 
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4.4.5 EXAMPLE SCENARIO 

An example of an actual scenario including an incorrect approach fix (LaGuardia A) 
is illustrated in this section. Other scenarios are documented in Appendix A. 

Figure 4.1 1 shows the subject’s aircraft was initially located over the Albany VOR 
at 18000 feet altitude, with a clearance to the New York / LaGuardia Airport via airway 

V157- Albany VOR to Kingston VOR to VALRE intersection to HAARP 
intersection to LaGuardia. For the first amendment (which was acceptable), at the 
GROUP intersection, the subject was cleared “direct Kingston vor, direct 
LaGuardia VOR; after LaGuardia expect vectors for ILS Runway 22 . This 

clearance was acceptable; however, if the subject rejected it, he was allowed to fly on the 
previous clearance via VI 57 as it made no difference in the scenario. 



VOR 

FIGURE 4.1 1. EXAMPLE SCENARIO - INITIAL CLEARANCE & FIRST CLEARANCE 

AMENDMENT 

NOT TO SCALE 


34 





The second clearance amendment, a clearance into weather, was given when the 
aircraft was 15 miles north of TRESA intersection (Figure 4.12). The amendment was 
given because of a change in the active runway, and the subject was notified by a change in 
the ATTS broadcast. 



FIGURE 4.12. EXAMPLE SCENARIO - WEATHER ERROR 
NOT TO SCALE 


Here, the subject was cleared “direct Pawling VOR, direct Bridgeport VOR, 
expect vectors for Localizer Runway 31.” This routed his aircraft directly into a 
line of thunderstorms stretching over much of the assigned pat h, a nd thus was 
unacceptable. If the subject accepted the clearance, he was allowed to penetrate the 
thunderstorm cell, and then at Pawling VOR, was given an acceptable third clearance 
amendment which routed him (as shown in Figure 4.13) via “direct Carmel vor, 
direct Deer Park VOR; after Deer Park expect vectors to Localizer Runway 
31; cross Deer Park at and maintain 5000 [foot altitude]; reduce speed 
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250 knots”. If the subject rejected the clearance, he was given essentially the same 



FIGURE 4.13. EXAMPLE SCENARIO - WEATHER ERROR RESOLUTION 
NOT TO SCALE 




The fourth clearance amendment, which contained an error, occurred when the 
aircraft ten miles north of the Deer Park VOR. The subject was cleared “direct dials 

intersection; descend and maintain 2500 [foot altitude] until 
established on the localizer for Localizer Runway 31; cleared for the 
Localizer Runway 31 approach” (Figure 4.14). 


LaGuardia / 
LOC Runway 31 


LaGuardia, 
VOR 


DIALS 
intersection 



Amendment / 

DIALS intersection, 
cleared for LOC 31 
(discontinuity created) 


Deer Park 
VOR 


Current Clearance / 1 
Deer Park VOR, 
vectors to LOC 31 


FIGURE 4.14. SCENARIO EXAMPLE - INCORRECT APPROACH FIX ERROR 
NOT TO SCALE 


The DIALS intersection was the initial approach fix for the Expressway Visual Approach to 
Runway 3 1 and was located to the southwest of the airport, well away from the extended 
centerline and Localizer approach for runway 3 1 . Regardless of his reaction to the 
clearance, as shown in Figure 4.15, he was eventually rerouted via a fifth clearance 
amendment “direct FABRY intersection ... cleared for the Localizer 
Runway 31 Approach, " with an acceptable localizer intercept angle. The scenario 
terminated at the FABRY intersection. 


37 




I ini' nil 


LaGuardia / 
LOC Runway 31 



LaGuardia, 
VOR 


DIALS A 

intersection FABRY 

intersection 


Deer Park 
VOR 


Resolution / 

FABRY intersection, | 
cleared for LOC 31 


FIGURE 4. 1 5. SCENARIO EXAMPLE - INCORRECT APPROACH FIX ERROR RESOLUTION 
NOT TO SCALE 
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4.5 data Collection and Scoring 

The data was recorded on video tape, software files, and observations sheets. Each 
scenario run was recorded by an 8mm video camera, focused on the simulator displays 
which also recorded ATC and intracockpit voice communications. In addition, the MIT 
Advanced Cockpit Simulator stored flight data and FMS information in a software archive 
for later analysis. Finally, observation sheets taken during each run by the PNF noted the 
outcome of each error, along with pilot comments and subjective ratings. Appendix B 
contains a sample observation sheet 

The observation sheets were the primary data source for the analysis, while the 
video and software archives were used as secondary sources for ambiguous observations. 
Observation sheets detailed each nominally unacceptable clearance error and the error 
detection performance of the pilot as initially, finally, or never detected. 

4.6 Ordering of Scenario Within and Between Subjects 

Because the test matrix was not rectangular, a Latin square-like arrangement altering 
procedure, mode of delivery, and scenario type could not be designed. As a practical 
matter, all runs which used the same mode of delivery were run contiguously in a block. 
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For example, ail verbal runs were accomplished, then all textual runs, and finally all 
graphical runs. To control for anticipatory effects, procedure and scenario were varied 
independently of each other so that sequential runs used different types of scenario. 

Finally, between subjects, the order in which the delivery mode blocks, scenarios, and 
procedures appeared was varied. Therefore, with the rotation scheme optimized for nine 
subjects, each element of the test matrix theoretically received three runs in each type of 
scenario, for a total of nine runs. Note that this yields 180 possible incorrect errors across 
all combinations of delivery modes and procedures (i.e. ten matrix elements, each with nine 
subject runs, with two unacceptable clearances per run). 

In the course of running the experiment, however, simulator anomalies, subject 
deviations from the required procedure, and scheduling difficulties led to a total of 163 
errors recorded. Of these, 83 were clearances into weather, and 80 were routing 
clearances. However, since the 28 illogical routing errors were ambiguous, the routing 
errors were analyzed on the basis of the remaining 52 clearances. Appendix C contains 
detailed information concerning delivery mode, procedure, and scenario rotation, as well as 
describing the data losses described above. 
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4.7 Subject information 


Nine male volunteer subjects, who were B-757/767 qualified air transport line 
pilots, participated in the study from August to October, 1992. All were from the same 
airline, with an average of 2583 hours in “glass cockpit” aircraft The Assistant Chief Pilot 
of the airline’s local domicile and the Air Line Pilots Association were contacted for 
assistance in recruiting subjects. Participants were based in the Boston area, where they 
were contacted by phone, and were reimbursed for their travel to MIT. Information on 
their experience level is summarized in Table 4.2. 


TABLE 4.2: SUBJECT EXPERIENCE SUMMARY 

Average Age 

48 Years 

Average Total Flight Time 

13338 Hours 

Average “Glass Cockpit” Flight Time 

2583 Hours 

Flight Qualification 

5 Captains / 4 First Officers 



5 Experimental results 


In order to evaluate the effect of delivery mode and procedure on situational 
awareness, it was first necessary to establish the overall ability of the subjects to detect the 
errors scripted into the scenarios. The difference in detection performance between weather 
and routing errors are presented in the first section. On the basis of these results, weather 
errors and routing errors were analyzed separately for delivery mode and procedure 
performance. 

To reiterate, initial detection means that the pilot rejected the amendment before 
readback, FMS programming, or accepting the clearance. If the pilot rejected the clearance 
after accomplishing any of these, the error was scored as a final detection. Finally, if the 
pilot was never aware that the clearance was nominally unacceptable, then the error was 
scored as never detected. Performance was evaluated statistically using a Non-Parametric 
Pairing Test, with 95% significance in the difference in performance used as the minimum 
level of statistical significance (see Appendix D for details of this method). 
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5.1 Weather & Routing Errors 

When examining pilot performance by type of error as shown in Figure 5.1, it was 
clear that weather errors were much more likely to be detected than routing errors. In the 
83 clearances issued into weather, 96% were detected at some point during the amendment 
procedure. Comparatively, only 55% of routing errors were detected by the subject pilots. 
In addition, most weather errors were detected initially (i.e. immediately upon receipt of the 
clearance amendment) when compared with routing errors. Both of these findings are 
statistically significant at the 99% level. 
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FIGURE 5.1. DETECTION OF WEATHER VS. ROUTING ERRORS 


Final 

Initial 


Based on the observations of the PNF, the subjects were aggressive in taking 
responsibility for weather separation. It may be that the simultaneous presentation of the 
weather radar reflectivity and navigation information on the EHSI enhanced the pilot’s 
ability to maintain weather separation, as the pilot could visually monitor whether the 
amended clearance he had just processed would route the aircraft through weather. 
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Conversely, routing errors appear to have been more difficult to detect This is 
possibly due to the fact that ATC often operates with routing or traffic constraints which are 
not apparent to individual aircrews. Furthermore, while thunderstorms are always a 
negative intrusion in an assigned clearance due to the possible degradation of safety, 
amended routings are a normal part of flight operations, and thus appear to be implicitly 
more subtle than weather errors. Because of the difference in detection performance, 
routing errors and weather errors were considered separately in the following analysis. 
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5.2 


EFFECT OF DELIVERY MODE 


5.2. 1 DELIVERY Mode performance for all errors 

In Figure 5.2, the error detection performance for all the errors is analyzed by 
delivery mode: there appears to be a trend favoring graphical over the textual and verbal 
mode. Pilots were able to detect 91% of all unacceptable clearances in the graphical mode. 
This is compares with only 78% for textual and 76% for verbal delivery (however, 
statistical significance of this effect cannot be shown because of the relatively small number 
of errors given). A statistically significant finding, in contrast, was that the vast majority of 
errors in the graphical mode were detected on initial reception of the clearance amendment 
as compared with sometime later during the procedure. 


100 



Verbal Textual Graphical 

Delivery Mode 


FIGURE 5.2. DETECTION OF ALL ERRORS BY DELIVERY MODE 
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5.2.2 Delivery Mode performance For Weather errors 


Strikingly, the subjects were able to detect 100% of the erroneous weather 
clearances immediately upon reception of the amendment with the graphical mode of 
delivery (Figure 5.3). This compared to 52% and 56% for the textual and verbal modes of 
delivery, respectively. This performance advantage of the graphical mode was confirmed 
by statistical analysis. In other words, pilots were immediately able to detect clearances 
into weather only in about half the situations when using the non-graphical delivery modes. 
In the graphical mode, because the aircraft's amended route was shown directly on the 
combination EHS I/weather radar display, it is possible that the subjects were able to 
immediately recognize conflicts between amended routings and thunderstorm cells. (It 
should be reiterated that almost all weather errors were detected at some point during the 
amendment process regardless of delivery mode, with a total of only 3 missed detections 
out of 83 errors.) 


■ Final% 

■ lnitial% 


Verbal Textual Graphical 

Mode 

FIGURE 5.3. DETECTION OF WEATHER ERRORS BY DELIVERY MODE 
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5.2.3 DELIVERY MODE PERFORMANCE FOR ROUTING ERRORS 

The graphical mode of delivery also appeared to have a slight advantage in the 
detection of unacceptable routing clearances as shown in Figure 5.4 (this result is 
significant when the graphical mode is compared with the verbal mode, but is not 
significant when graphical mode is compared with the textual mode because not enough 
data were collected). However, as in other cases, the graphical mode excelled in detections 
during the initial review of the information by the pilot; this result is statistically significant. 
A possible reason for this is that, because graphical clearances are displayed on the EHSI, 
they can be direcdy compared with the current routing. 
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FIGURE 5.4. DETECTION OF ROUTING ERRORS BY DELIVERY MODE 


5.2.4 DELIVERY MODE SUMMARY 

In summary, the graphical mode appears to outperform both the verbal and textual 
modes of delivery in two areas. First, subjects appeared to detect a greater percentage of 
erroneous routing clearances with the graphical mode. Second, there is a strong indication 
that graphical presentation is useful in helping pilots detect errors more quickly than either 
of the other two modes. While the quickness in evaluating clearances with the graphical 
mode is not sufficient by itself to recommend graphical delivery, there may be some 
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subjective benefit to the pilot if graphical delivery aids the timeliness of decision making. 
Thus, there appears to be a general basis for recommending graphical delivery of clearance 
amendments. 

5.3 Effect of Procedure 

Because of the construction of the test matrix, manual FMS programming without 
readback (procedure 3) cannot be statistically compared with the other procedures. It is 
included in the performance graphs for qualitative comparison only. 

5.3.1 Procedure Performance for weather errors 

Vectors into weather did not yield any statistically significant differences between 
the procedures because of the high detection rate (near 100%) for all procedures (Figure 
5.5). Furthermore, initial detection of weather events also does not appear to change with 
procedure. 
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FIGURE 5.5. DETECTION OF WEATHER ERRORS BY PROCEDURE 
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5.3.2 PROCEDURE PERFORMANCE FOR ROUTING ERRORS 


For the routing errors, it appears that the procedures with automated programming 
may have yielded better detection performance than those with manual programming 
(Figure 5.6). When the individual procedures are combined, automated FMS 
programming yielded an aggregate detection percentage of 64%, while the procedures with 
manual programming combined for only 42%. In contrast, the effect of readback was 
inconsistent, with readback appearing to help in the automated FMS programming cases 
and hinder when the FMS was manually programmed. (Insufficient data was collected to 
confirm these trends statistically). 


100 


1 

o 

0) 

80- 

0) 

o 

60- 

CO 


o 

k. 

40- 

in 


o 

20 - 

sP 

o- 



Procedure 


FIGURE 5.6. DETECTION OF ROUTING ERRORS BY PROCEDURE 
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5.3.3 procedure Summary 

Despite seeming to have a more involved level of review, manual programming 
appears to be a less desirable for situational awareness. A possible reason for this is that 
the type of cognitive processing required to program the FMS appeared to reduce the pilot’s 
overall ability to evaluate on a strategic level. In support of this, it was clear from the PNF 
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observations that the clearance amendments processed with manual programming required 
much more time to execute compared to those with automated programming. This indicates 
that automated FMS programming may be desirable in that it possibly allows pilots to 
spend more of their cognitive time and resources at the strategic level thereby potentially 
increasing situational awareness. However, as stated previously, there may be a bias 
against manual FMS programming detection performance because the simulator CDU 
differs from the unit used in actual flight operations. 

While the data indicate no overall trend in the value of readback, it appears that the 
combination of readback with automated programming may be beneficial to situational 
awareness. Unfortunately, there was not enough data to confirm this trend. A possible 
explanation for this effect may be that readback inspires additional scrutiny of the clearance 
amendment which might not occur with automated FMS programming alone. Further 
study regarding this possible effect is warranted. 
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6 SUBJECTIVE RESULTS 


It should be noted that because only nine subjects participated in this experiment, 
caution should be exercised in the interpretation of the results in this section due to the 
small sample size. 

6.1 Subjective Ratings of individual Format/Procedure 
Combinations 

Subjective ratings were taken after each experimental run for each format/procedure 
combination. The subjects were asked for evaluations of overall effectiveness, situational 
awareness, and time efficiency of the combination of display mode and procedure just 


flown. 



6.1.1 Ratings for Overall effecti venes s 


The subjective ratings for overall effectiveness ratings are presented in Figure 6.1. 
There was an apparent subjective preference for datalink. With only one exception, all 
procedures with textual and graphical delivery were rated higher than verbal modes, and 
every procedure which required automated FMS programming rated higher than the current 
ATC procedure (i.e. verbal with readback and manual FMS programming). The most 
highly rated procedure was textual presentation with automated programming and 
readback, with graphical delivery with automated programming and no readback close 
behind. This implied that, on average, pilots desired readback with textual delivery, but 
that graphical was preferred without readback. (Note tha t this does not contradict the 
performance data. In the graphical mode of delivery with automated FMS programming, 
error detection performance was similar with or without readback.) 
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FIGURE 6. 1 . SUBJECTIVE RATINGS OF OVERALL EFFECTIVENESS OF DELIVERY 
MODE/PROCEDURE COMBINATIONS 
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6.1.2 Ratings For Situational awareness 


Similarly, datalink did not imply any perceived loss in the situational awareness 
ratings, either with automation or with delivery mode (Figure 6.2). With only one 
exception, the graphical and textual modes of delivery outrated the verbal mode within the 
same procedure. Also, the subjects indicated that the net effect of automated FMS 
programming would be beneficial to situational awareness when compared to the current 
procedure. Some subjects commented that they would be able to devote more time to 
evaluating the content of new clearances with automated programming, rather than 
accomplishing data entry tasks on the FMS, thereby increasing their perceived situational 
awareness. Readback, to a lesser extent, was also seen as an enhancement to situational 
awareness. The subjects on average rated procedures with readback higher than those 
without readback within the same method of FMS programming. (An exception to this 
occurs with automated FMS programming and graphical delivery. A possible reason for 
this is discussed in Section 7.4.) 
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FIGURE 6.2. SUBJECTIVE RATINGS OF SITUATIONAL AWARENESS OF DELIVERY 
MODE/PROCEDURE COMBINATIONS 
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It should also be noted that procedures, regardless of mode of delivery, were 
generally rated similarly for situational awareness by subjects. That is, all cases with 
automated FMS programming and readback were rated higher than those with other 
procedures, while manual FMS programming with no readback rated lowest. 

6.1.3 Ratings for time efficiency 

In terms of time efficiency all of the procedures with automated FMS programming 
oistrated those with manual programming (Figure 6.3). However, there seems to be no 
strong correlation between time efficiency and the requirement for readback. Additionally, 
textual presentation on average was perceived as an expeditious mode of delivery. Some 
subjects indicated that this was due to the compact nature of the textual display. Graphical 
delivery, as some subjects commented, required them to search different displays to find all 
of the clearance information. Note that this is a result of the particular implementation of 
the graphical mode used in this experiment, and may or may not apply to other 
implementations. Other subjects also stated that verbal delivery was more time consuming, 
as the clearance needed to be written down on paper. 



(1 = unsafe, 4 = comparable to current systems, 7 = optimal) 


FIGURE 6.3. SUBJECTIVE RATINGS OF TIME EFFICIENCY OF DELIVERY MODE/PROCEDURE 
COMBINATIONS 
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6.1.4 SUMMARY OF SUBJECTIVE INDIVIDUAL FORMAT/PROCEDURE COMBINATION 
RATINGS 

In summary, the subjective ratings indicated that pilots appeared to prefer automated 
programming in terms of time efficiency, situational awareness, and overall effectiveness. 
Pilots also liked the textual and graphical presentations of clearance amendments, 
particularly in overall effectiveness. Automated programming was not felt to degrade 
situational awareness, and in fact was perceived as an enhancement, while the perceived 
benefit of readbacks depended on the delivery mode and method of FMS programming. 
Finally, the currently used procedure of manual FMS programming with readback rated 
nearly last in all three categories. 



6.2 Post Experiment Summary Ratings 


After experiencing the entire test matrix, subjects were asked to summarize their 
opinions of the delivery modes and procedures during an exit interview. They were asked 
both to rank (i.e. order from most desirable to least desirable) and to rate (i.e. assign a 
numeric value from 1 (lowest) to 10 (highest)) each delivery mode and procedure. 

6.2. 1 Delivery Mode Rankings & ratings 


In terms of the delivery modes, pilots were evenly split between textual and 
graphical modes overall in terms of ranking, with each receiving 4 top rankings. However, 
graphical received a slightly higher numerical rating, thus implying that graphical may 
possibly have additional benefits over textual delivery. (Figure 6.4). For instance, subjects 
made the comment that graphical was preferable for quickness in evaluation of the 
clearance, but they also stated that details such as crossing restrictions or runway changes 
were often difficult to discern. Conversely, textual mode was praised for being compact 
and accurate, but was criticized for having few decision-making advantages over voice. 




FIGURE 6.4. EXIT INTERVIEW RANKINGS & RATINGS OF DELIVERY MODES 
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6.2.2 PROCEDURE RANKINGS & RATINGS 


When asked to rate procedures, pilots chose the procedures with automated 
programming highest overall (Figure 6.5). Pilots commented that automated programming 
allowed for more “heads up” time, or time to concentrate strategic aspects of the clearance. 
Conversely, there was not a definite preference for readback. The procedure which was 
ranked and rated highest by most subjects was automated programming without readback. 
However, the standard deviation of 2.85 for this procedure was the highest among all 
procedures, indicating a wide range of opinion on its overall value. Readback with 
automated programming, in contrast, was consistently rated high by the pilots (with the 
lowest standard deviation of 0.845). 



Procedure 

No Readback / 
Automated Programming 

Readback / 

Automated Programming 

No Readback / 

Manual Programming 

Readback / 

Manual Programming 
(Current Procedure) 


Error Bar = 1 Standard Deviation 


6 5 4 3 2 1 0 

Number of Highest Subjective 
Rankings 



i ■ i 1 r 
1 2 3 4 5 6 

Mean Subjective Rating 
(1 s lowest, 10 = highest) 


FIGURE 6.5. EXIT INTERVIEW RANKINGS & RATINGS OF PROCEDURES 


6.2.3 SUMMARY OF POST EXPERIMENT RANKINGS & RATINGS 

In summary, graphical and textual delivery were given similar subjective scores, 
with each perceived as having different advantages and disadvantages. Graphical delivery 
was praised for its decision-making advantage, while textual delivery s benefit was its 
compact format. Automated programming was almost unanimously endorsed by the 
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subjects. A definite conclusion about readback was not indicated, but there may be benefits 
for its retention in clearance amendment procedures with automated FMS programming. 
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7 ADDITIONAL DISCUSSION 


In addition to the performance data and subjective ratings, subjects were 
encouraged to comment about issues which they felt were important about datalink. This 
section discusses some of the points to be addressed in the implementation of datalink 
delivery of ATC clearances as originated by the subjects. 

7.1 Combined Textual and Graphical presentation 

As stated above, subjects were divided about the perceived advantages between the 
graphical and textual modes of delivery. In particular, subjects often gave contradictory 
comments about the graphical mode of delivery. Some subjects stated that with the 
graphical mode, it was sometimes difficult to understand the clearance fully because the 
information was associated with many different symbols and displays in the cockpit Text 
(as some subjects stated) instead placed all of the clearance information in a single location 
on the dedicated datalink textual display. Nevertheless, most did like the graphical mode 
because they were able to see on the EHSI where they were cleared to immediately upon 
receiving the clearance. 

Because of this, eight of the nine subject pilots supported the concept of delivering 
the information using both textual and graphical simultaneously, thus combining the 
advantages of both modes. In fact, several subjects independently suggested this during 
the experiment. One subject stated, “text with graphical ... gives redundancy like a 
readback”. In fact, in the absence of getting both text and graphical simultaneously, 
another pilot did not want to switch from using the voice frequency at all. 
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7.2 Value of Readback 


The net effect of readback as pan of the clearance amendment procedure is not 
clear. While all subjects felt that verbal delivery necessitated a readback as an error 
checking procedure, they were unable to provide an consensus on its value in the textual 
and graphical modes of delivery. However, there appeared to be some value in combining 
automated FMS programming with readback. 

In terms of error detection performance, readback appeared to be beneficial when 
combined with automated programming, but insufficient data was accumulated to 
substantiate this trend statistically. The subjective situational awareness scores indicated 
that automated programming with readback rated higher than any other procedure, 
regardless of delivery mode. Finally, in the exit interview, automated programming with 
readback was ranked first or second by every subject and furthermore was given a 
consistently high subjective score. 

Taken together, there appears to be an indication for the use of readback coupled 
with automated programming in any future procedure for clearance amendments using ATC 
datalink. Note that it would not necessarily be required to readback the clearance to ATC; it 
could be maintained as an intracockpit crew procedure. 

7.3 Textual Display Implementation Issues 

While the textual display used in the experiment was generally thought to be 
adequate, pilots did have some comments about its implementation, particularly if the 
datalink system is retrofitted to existing non-EFIS aircraft. In particular, some subjects 
were concerned about the readability and placement of the textual screen. Other subjects 
mentioned that textual displays (and graphical displays, to some extent) may be susceptible 
to turbulence (i.e. the vibration in turbulence would make the displays unreadable). In 
general, usage of the visual modes of delivery elicited comments about the potential 
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problem of too much “heads down” time, or time spent looking at instruments in the 
cockpit rather than visually scanning the airspace surrounding the aircraft. Other comments 
indicated that, as pilots get older, hyperopia (farsightedness) may become a problem, and 
that reading small text in the near visual field may become difficult 

Finally, the format used for the textual display in this experiment was based on a 
simple transcription of the current ATC lexicon. However, more optimal formats may exist 
which should be investigated. One subject suggested doing away with the standard 
phraseology, and instead display the textual routing information in a format similar to that 

of existing FMS displays. 

7.4 GRAPHICAL DISPLAY IMPLEMENTATION ISSUES 

As demonstrated by the relatively low situational awareness rating of graphical 
mode and automated FMS programming without readback, implementation issues need to 
be addressed concerning graphical display of clearance amendment information. In 
particular, care should be taken to insure that details of the clearance amendment are 
difficult to overlook. These include; method of annunciation, symbology, color, 
distribution of information on the flight deck, and the presentation of detailed and/or 
complex information. In particular, it should be noted that certain complex ATC 
procedures were not included in this experiment; for example, speed adjustments which 
would take place upon reaching an assigned altitude rather than a particular location. 

In addition, the implementation used in this simulation study distributed information 
to different graphical displays to provide a more natural context Advantages and 
disadvantages to distributing the information to different flight displays may exist. For 
instance, while it may aid cognitive processing to locate an altitude assignment on the 
altitude tape display, it may hinder interpretation of the clearance because this information is 
not grouped with the remain clearance data. Conversely, grouping all the clearance 
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information on one display may be confusing. Further investigation of the effects of this 
distribution, as well as the optimal method of partitioning data, should be undertaken. 

However, because particular care was taken to avoid the presentation of any textual 
information with the graphical amendment (thus preventing a confounding variable), 
designers of graphical clearance amendment displays may have more flexibility available to 
them. Finally, care should be taken to ensure that the implementation of graphical clearance 
amendment delivery does not contribute to “cluttered” navigational displays. 
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8 CONCLUSIONS AND RECOMMENDATIONS: 

While recommendations can be made on the basis of these results, it should be 
reiterated that because of the small sample size, the statistically significant results are 
limited. Also, because the subject pool consisted of volunteers, the sample population may 
be biased towards favoring new technology when compared with the mean pilot 
population. 

Graphical Mode Yielded Best Error Detection Performance 

The graphical mode yielded the best performance in detecting nominally 
unacceptable clearances. In addition to the best overall performance, it also had the 
advantage that the vast majority of errors were detected rapidly upon initial review of the 
clearance. In addition, graphical was outstanding in the detection of vectors into weather, 
with 100% of all errors detected immediately upon receiving the information. This may 
illustrate the possible benefit of having a display which is common to both clearance 
amendments and potential sources of hazard or conflict. However, there was some 
indication from pilot comments that a purely graphical mode of delivery required additional 
study to ensure a beneficial implementation. 

Combined Textual & Graphical presentation Desired 

The primary advantage of graphical delivery was in the rapid evaluation of 
clearances. However, because graphical implementation required that information was 
distributed to many different displays within the cockpit, this made certain details in the 
clearance amendment difficult to read. Textual delivery has the advantage of having all the 
information in one place in a concise format. Nevertheless, textual delivery seems to have 
few decision-aiding advantages over the current verbal delivery. It seems likely that a 
simultaneous presentation in both text and graphics will combine the advantages of the 
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individual modes, and eight of nine subject pilots desired this. This is also consistent with 
the current dual representation of information in existing FMS/EFIS Systems. Further 
investigation into this possible “mixed” delivery mode is warranted. 

Automated Programming Yielded Better Error detection 
Performance 

Pilots appeared to be able to detect more unacceptable amendments with the use of 
automated FMS programming compared to manual programming. Additionally, the 
majority of pilots desired automated programming with datalink. It is also clear from 
subject comments that manual programming does not appear to help pilots understand the 
overall implications of accepting a particular clearance amendment. In contrast, automated 
programming appears to allow the pilot to concentrate on evaluating the clearance on the 
strategic level. It is recommended that automated FMS programming technology be 
considered for use with datalink delivery of ATC clearances. 

Readback With automated Programming May Be Beneficial 

The effect of readback on detection performance appeared to be dependent upon the 
method of FMS programming. Although there is no statistical significance to the error 
detection performance, readback combined with automated programming appeared to yield 
somewhat better performance. Additionally, pilots on average rated the procedures with 
readback and automated FMS programming higher than any other procedure in terms of 
subjective situational awareness. Taken together, there is an indication that readback may 
have a benefit when used with automated FMS programming. While it is recommend that 
readback be retained on this basis, further study on the effectiveness of readback is 
warranted ... 
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APPENDIX A: 


SCENARIOS 


This appendix contains complete documentation of the scenarios developed for 
use with this simulation study. These scenarios were designed for the 30 May 1991 
edition of the US Government Enroute Low Altitude IFR Charts, specifically Charts L- 
21, L-23, L-25, and L-27. Table 4.1 describes general information about each scenario. 
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Scenario Boston A 

The aircraft will start at ALBANY VOR. The initial clearance is: ALBANY, direct 
GARDNER VOR, direct REVER intersection. ATIS Charlie is active. 

Amendment Series 1 : Routing into WX 

Reference the ATC station to GRAVE intersection. 

When the aircraft reaches GRAVE, do the following: 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #1 

YOU: "(callsign), direct BRADLEY VOR, Victor 4-l-9er BOSOX 

intersection. After BOSOX, expect vectors to ELS Runway 4 Right." 


IF THE PILOT UQES. NQ1 REJECT THE AMENDMENT: 

reference the ATC station to CHESTER VOR (CTR). When the aircraft 
reaches CHESTER. 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #2 
YOU: "(callsign), direct FAIDS intersection, direct BOSOX 
intersection. After BOSOX, expect vectors to ILS Runway 4 Right. 
Cross BOSOX at and maintain seven thousand feet and 2-1-0 knots." 

IF THE PILOT DQES REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENTS 

YOU: "(callsign), direct FAIDS intersection, direct BOSOX 

intersection. After BOSOX, expect vectors to ELS Runway 4 Right. 

Cross BOSOX at and maintain seven thousand feet and 2-1-0 knots." 


Amendment Series 2: OK 

Reference the ATC station to FAIDS intersection . 

When the aircraft reaches FAIDS, do the following: 
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YOU: "(callsign), clearance amendment" 
when the pilot is ready: 

ARM AND SEND AME NDMENT #3 

YOU: "(callsign), direct MILIS intersection. After MILIS expect vectors for 
ILS RWY 4 right. Cross MILIS at and maintain 7000 and 2-1-0 
knots." 

Since this amendment is OK, Don 't expect anything to change. If he rejects the 
amendment, let him continue on his current amendment. Otherwise, stall him until 
BOSOX and go to the next amendmerU series. 

Amendment Series 3: Bad waypoint 

Reference the ATC station to BOSOX intersection. 

When the aircraft reaches BOSOX, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SE ND AMENDMENT #4 

YOU: "(callsign), direct COHAS intersection. Descend and maintain 3000 
until established on the localizer for ILS runway 4 right approach." 
AT THIS POINT. VERBALLY CLF -AR HIM FOR THE APPROACH 


IF THE PILOT UQES. NOT REJECT THE AMENDMENT: 

reference the ATC station to NABBO. When the aircraft is about 5 miles 
from NABBO. 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AM ENDMENT #5 

YOU: "(callsign), direct NABBO intersection. Maintain 3000 until 
established on the localizer for ILS runway 4 right approach." 

AT THIS POINT. VERBALLY CLEAR HIM FOR THE 

APPROACH 

IF THE PILOT DOFS REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AM ENDMENT #5 

YOU: "(callsign), direct NABBO intersection. Maintain 3000 until 
established on the localizer for ILS runway 4 right approach." 

AT THIS POINT. VE RBALLY CLEAR HIM FOR THE 
APPROACH 
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This sc enario will terminate at NABBO. 

A77S-C: 

"Boston Logan Airport information Charlie. 2050 Zulu. 5000 Scattered Est 

Ceiling 10000 broken, visibility 5 haze. Temp: 97 Dewpoint: 96. Wind 
calm. Altimeter 28.85. Lightning reported W of airport. Approach ILS 
RWY 4R, ILS/DME 33L. Departing RWY 4L, 9. CONVECTIVE 
SIGMET NOVEMBER 5 in effect; contact FSS for details. Numerous 
Cranes and other construction equipment southwest of airportAdvise on 
initial contact information Charlie" 

CONVECTIVE SIGMET NOVEMBER 5 

Lines of heavy thunderstorms extending from 10 mi E CON to 30 mi E BDR to 
10 mi W ACY to 20 mi E SLT to 10 S of SLK to 10 mi E CON reported 
moving NE at 25 knots, with tops reaching FL450 or greater. Severe 
turbulence and hail reported below 10000 feet, with moderate icing above 
15000 feet. 




Scenario BWI A 


The aircraft will start at HARBO intersection. The initial clearance is: HARBO, V- 
268 BALTIMORE VOR. ATIS Papa is active. 


Amendment Series 1: Routing into WX 
Reference the ATC station to AVALO intersection. 

When the aircraft is 20 NMifrom AVALO, do the following: 


YOU: " (callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SF NH AMENDMENT #1 

YOU: "(callsign), direct WATERLOO VOR, Victor 3-0-8 CHOPS 

intersection. After CHOPS, expect vectors to ILS Runway 3-3 Left. 

Note: ATIS Quebec now active 


IF THE PILOT DOES NOT REJECT THE AMENDMENT: 

reference the ATC station to WATERLOO VOR (ATR). When the 
aircraft reaches WATERLOO: 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM ANT) SEND AM ENDMENT #2 

YOU: "(callsign), direct BALTIMORE VOR. Expect vectors to ILS 
Runway 3-3 Left. Descend and maintain eight thousand. Reduce 

speed 2-5-0 knots." 

IF THE PILOT DOES. REJECT THE AMENDMENT: 

YOU: " (callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT # 3 

YOU: "(callsign), direct SEA ISLE VOR, Victor 4-4 AGARD 
intersection. After AGARD, expect vectors to ILS Runway 3-3 left, 
cross AGARD intersection at and maintain one-zero thousand. 
Reduce speed 2-5-0 knots." 


Amendment Series 2: OK 
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Reference the ATC station to CANNY intersection. 

When the aircraft reaches CANNY, do the following: 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #4 

YOU: "(callsign), direct PALEO intersection. after PALEO expect vectors 
to ILS RWY 33L; cross PALEO at and maintain 8000 and 2-1-0 
knots" 

This amendment is OK, but if the pilot rejects it, let him maintain current clearance. If he 
requests something weirdy stall him till AGARD and tjhe next amendment series^ 

Amendment Series 3: Bad waypoint 

Reference the ATC station to AGARD intersection. 

When the aircraft reaches AGARD, do the following: 

YOU: "(callsign), clearance amendment" 

when the pilot responds ready: 

ARM A N D. SE N D AMENDMENT « 

YOU: "(callsign), direct B ALTO intersection. Descend and maintain 3000 
until established on the localizer for ILS runway 3-3 left." 

AT THIS POINT. VERBALLY CLEAR HIM FOR THE APPROACH 


IF THE PILOT DOES NOT REJECT THE AMENDMENT: 

reference the ATC station to SPLAT. When the aircraft is about 5 miles 
from SPLAT: 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #6 

YOU: "(callsign), direct SPLAT intersection. Maintain 3000 until 
established on the localizer for ILS runway 3-3 Left." 

AT THIS POINT. VERBALLY CLEAR HIM FOR THE 

APPROACH 

IF THE PILOT POES REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT #6 

YOU: "(callsign), direct SPLAT intersection. Maintain 3000 until 
established on the localizer for ILS runway 3-3 Left." 
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AT THIS POINT. V F.RB ALLY CLEAR HIM FOR THE 
APPROACH 

This s cenario will re,rn ^ nate at SPLAT. 

ATIS-P: 

"Baltimore Washington International Airport information Papa. 1650 Zulu. 7000 
Scattered Est Ceiling 15000 broken, visibility 10. Lightning reported N of 
airport. Temp: 90 Dewpoint: 81. Wind 130 at 5. Altimeter 29.17. Visual 
Approach RWY 15R. Departing RWY 10. Windshear advisories in effect. 
Bird Activity east of airport. Advise on initial contact information Papa 

ATIS-Q: 

"Baltimore Washington International Airport information Quebec. 1723 Zulu. 
2000 Scattered, Measured Ceiling 7000 overcast, visibility 3 miles light 
rain, occassional 1000 overcast, visibility 1/2 in thunderstorms. Temp. 82 
Dewpoint: 80. Wind 310 at 20 gusting to 28. Altimeter 28.82. Approach 
ILS RWY 28 and ILS RWY 33L, Departing RWY 28. Windshear 
advisories in effect Advise on initial contact information Quebec 
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Scenario Kennedy A 

The aircraft will start at Sandy Point VOR (SEY). The initial clearance is: SEY, V- 
268 ERICK intersection, direct JFK VOR. ATIS Kilo is active. 

Amendment Series 1 : OK 

Reference the ATC station to HAMPTON VOR (HTO) intersection. 

When the aircraft is 25 miles from HTO, do the following: 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready : __ 

ARM AND SEN D AMENDMENT#! 

YOU: "(callsign), direct HAMPTON VOR, direct ERICK 

INTERSECTION, direct JFK. Expect vectors to ILS Runway 3-1 
right." 

This amendment is OIK Allow deviations within reason . 

Amendment Series 2: Routing into WX 
Reference the ATC station to ERICK intersection. 

When the aircraft reaches ERICK, do the following: 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #2 

YOU: "(callsign), direct DEER PARK VOR. After DEER PARK, expect 
vectors to ILS Runway 3-1 right." 


IF THE PILOT UQES. NDL REJECT THE AMENDMENT: 

Wait about 1 minute: 

YOU: "(callsign), sorry to do this to you, but LaGuardia got some of 
our airspace and we have another clearance amendment for you" 

when the pilot responds ready: 

ARM AND SEND AMENDMENT S 

YOU: "(callsign), direct SHIPP intersection. After SHIPP expect 
vectors to ILS Runway 3-1 right. Cross SHIPP at and maintain 5 
thousand. Reduce Speed 2-5-0 knots" 
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IF THE PILOT DOFS REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT #3 

YOU: "(callsign), direct SHIPP intersection. After SHIPP expect 
vectors to ILS Runway 3-1 right Cross SHIPP at and maintain 5 
thousand. Reduce Speed 2-5-0 knots" 

Amendment Series 3: Bad waypoint 
Reference the ATC station to SHIPP intersection. 

When the aircraft is five miles from SHIPP, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEN D AMENDMENT #4 

YOU: "(callsign), direct NARRO intersection. Descend and maintain 3000 
until established on the localizer for ILS runway 3-1 right." 

AT THIS POINT. VERBALLY C LEAR HIM FOR THE APPROACH 


IF THE PILOT QQES. NQ1 REJECT THE AMENDMENT: 

reference the ATC station to LORAC. When the aircraft is about 5 miles 
from LORAC: 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENTS 

YOU: "(callsign), direct LORAC. Maintain 3000 until established on 
the localizer for the ILS runway 3-1 right approach." 

AT THIS POINT. VF RB ALLY CLEAR HIM FOR THE 

APPROACH 

IF THE PILOT DQES. REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AM ENDMENT #5 

YOU: "(callsign), direct LORAC. Maintain 3000 until established on 
the localizer for the ILS runway 3-1 right approach." 
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AT THIS POINT. VERBALLY CL EAR HIM FOR THE 

APPROACH 

This s cenario will terminate at LOR AC. 

ATIS-K ; 

"John F Kennedy International Airport information Kilo. 1945 Zulu. 5000 

Scattered Est Ceiling 2-5000 broken, visibility 15. Temp: 76 Dewpoint: 

64 Wind 300@7. Altimeter 29.45. Lightning reported E of airport. 
Aircraft Landing and departing 31L, 31R. CONVECTIVE SIGMET 
DELTA THREE in effect; contact New York FSS for details. Advise on 
initial contact information Kilo" 

CONVECTIVE SIGMET DELTA 3 

Line of heavy thunderstorms extending from 50 mi N CAR to 30 mi E CON to 10 
mi E BDR to 20 mi W BDR to 10 mi W of BML to 50 mi N CAR reported 
moving SE at 20 knots, with tops reaching FL300. Severe turbulence and 
hail reported below 10000 feet, with moderate icing above 15000 feet. 


73 



Scenario LaGuardia A 


The aircraft will start at ALBANY VOR. The initial clearance is: ALBANY, V-157 
LAGUARDIA VOR. AT IS Foxtrot is active. 

Amendment Series 1 : OK 

Reference the ATC station to GROUP intersection. 

When the aircraft is at GROUP, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT!! 

YOU: "(callsign), direct KINGSTON VOR, direct LAGUARDIA VOR. 
After LAGUARDIA, expect vectors to ILS Runway 2-2." 

This amendment is OK, and he should accept it. However, if he does not, let him fly the 
old route. IJjiot, staU him to WIG AN -5 and amendment series 2. 

Amendment Series 2 : Routing into WX 
Reference the ATC station to TRESA intersection. 

When the aircraft is 15 miles from TRESA, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEN D AMENDMENT #2 

YOU: "(callsign), direct PAWLING VOR, direct BRIDGEPORT VOR. 
After BRIDGEPORT, expect vectors to Localizer Runway 3-1." 

Note: ATIS Golf now active. 


IF THE PILOT DOES NOT REJECT THE AMENDMENT: 

reference the ATC station to PAWLING VOR (PWL). When the aircraft 
reaches PAWLING: 

YOU: "(callsign), clearance amendment, " 

when the pilot responds ready: 

ARM AND SEND AM ENDMENT #3 

YOU: "(callsign), direct CARMEL VOR, direct DEER PARK VOR. 
After DEER PARK expect vectors to Localizer Runway 3-1. Cross 
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DEER PARK at and maintain five thousand. Reduce Speed 2-5-0 
knots." 

IF THE PILOT UQES. REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT #4 

YOU: "(callsign), direct KINGSTON VOR, direct DEER PARK 
VOR. After DEER PARK, expect vectors to Localizer Runway 3-1. 
Cross DEER PARK at and maintain five thousand. Reduce Speed 2- 
5-0 knots." 


Amendment Series 3: Bad waypoint 

Reference the ATC station to DEER PARK (DPK) VOR. 

When the aircraft is ten miles from DPK, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #5 

YOU: "(callsign), direct DIALS intersection. Descend and maintain two 

thousand, five hundred until established on the localizer for Localizer 
runway 3-1." 

AT THIS POINT. VERBA LLY CLEAR HIM FOR THE APPROACH 


IF THE PILOT DOFS NOT REJECT THE AMENDMENT; 

reference the ATC station to FABRY. When the aircraft is about 5 miles 
from FABRY 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #6 

YOU: "(callsign), direct FABRY. Maintain two thousand, five 
hundred until established on the localizer for Localizer runway 3-1." 
AT THIS POINT. VERBALLY CLEAR HIM FOR THE 
APPROACH 

IF THE PILOT DOFS REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 
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ARM AND SEND AMENDM ENTS 

YOU: "(callsign), direct FABRY. Maintain two thousand, five 
hundred until established on the localizer for Localizer runway 3-1. 

AT THIS POINT. VERBA LLY CLEAR HIM FOR THE 

APPROACH 

This s cenario will terminate^ at FABRY. 

ATIS-F: 

"LaGuardia Airport information Foxtrot. 2150 Zulu. 10000 Scattered, Est 

Ceiling 20000 broken, visibility 20. Lightning reported NE of airport. 
Temp: 75 Dewpoint: 68. Wind 180 at 15. Altimeter 29.14. ILS and 
visual Rwy 22 approaches in effect. Windshear advisories in effect. Bird 
Activity south of airport. Advise on initial contact information Foxtrot" 

ATIS-G: 

"LaGuardia Airport information Golf. 2224 Zulu. 3000 Scattered, Est Ceiling 

6000 overcast, visibility 3 miles light rain. Temp: 70 Dewpoint: 68. Wind 
300 at 20 gusting to 28. Altimeter 29.02. Arrivals expect Expressway 
Visual and Localizer Rwy 31. Convective SIGMET Delta in effect for 
New Jersey and Southeast New York. Contact NewYork FSS for details. 
Windshear advisories in effect. Bird Activity reported south of airport. 
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Scenario Boston B 


The aircraft will start near HAMPTON VOR (HTO). The initial clearance is: 

HAMPTON, victor-139 PROVIDENCE VOR, direct BOSTON VOR. ATIS Juliet is 

active. 

Amendment Series 1 : OK 

Reference the ATC station to HAMPTON VOR. 

When the aircraft reaches HAMPTON, do the following: 


YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: f 
ARM AND SEND AMENDMENT #1 

YOU: "(callsign), direct BOSTON VOR, expect vectors to ILS Runway 4 
right." 

This amendment is fine. There should be no reason not to accept it. If he does, roll your 
eyes and have him maintam greseru clearance^ 

Amendment Series 2: Routing into WX 

Reference the ATC station to TRAIT intersection. 

When the aircraft reaches TRAIT, do the following: 


YOU: "(callsign), clearance amendment, " 
when the pilot responds re ady: 

ARM AND SEND AMENDMENT #2 


YOU: 


"(callsign), direct DRUNK intersection, direct TONNI intersection. 
After TONNI, expect vectors to ILS Runway 2-7. Cross TONNI at 
and maintain 50(H) and 2-1-0 knots" 

Note: ATIS Kilo now active 



IF THE PILOT UQES. NOT REJECT THE AMENDMENT: 

reference the ATC station to LAFAY interesection When the aircraft 
reaches LAFAY: 

YOU: "(callsign), clearance amendment, " 

when the pilot responds ready: 

ARM AND SEND AMENDMENT #3 

YOU: "(callsign), direct PROVIDENCE VOR, direct PUTNAM 
VOR, after PUTNAM, expect vectors to ILS Runway 2-7." 
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IF THE PILOT DOES. REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEN D AMENDMENT #3 

YOU: "(callsign), direct PROVIDENCE VOR, direct PUTNAM 
VOR, after PUTNAM, expect vectors to ILS Runway 2-7." 

Amendment Series 3: Illogical routing 

Reference the ATC station to PROVIDENCE VOR (PVD). 

When the aircraft reaches PVD, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT M. 

YOU: "(callsign), direct HARTFORD VOR, direct BOSTON VOR. After 
BOSTON, expect vectors to ILS RWY 2-7. Cross BOSTON at and 
maintain 7000 and 2-1-0 knots." 


IF THE PILOT DOES. NOT REJECT THE AMENDMENT: 

reference the ATC station to FOSTY. When the aircraft is at FOSTY: 
YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #5 

YOU: "(callsign), direct BOSTON VOR. Expect vectors to ILS 
runway 2-7. Cross BOSTON at and maintain 7000 and 2-1-0 knots.' 

IF THE PILOT DOES. REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT #5 

YOU: "(callsign), direct BOSTON VOR. Expect vectors to ILS 
runway 2-7. Cross BOSTON at and maintain 7000 and 2-1-0 knots. | 

This s cenario wil[ terminate at BOS. 

ATIS-J: 

"Logan airport information Juliet. 1947 Zulu. 4000 Scattered, Est Ceiling 10000 
Broken, visibility 8 miles. Temp: 88 Dewpoint: 75. Wind calm. Altimeter 
29.42. Visual approach RWY 4R, Rwy 33L in effect Departing RWY 
4R, 4L. Windshear advisories in effecL Numerous Cranes and other 
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construction equipment located southwest quadrant of airport. Advise on 
initial contact information Juliet" 

ATIS-K: 

"Logan airport information Kilo. 2014 Zulu. Measured Ceiling 2500 Broken, 
6000 Overcast, visibility 3 miles in light rain showers Temp: 77 
Dewpoint: 75. Wind 280 at 18 gusting to 25. Altimeter 29.12. ILS DME 
approach RWY 27 in effect. Windshear advisories in effect. Numerous 
Cranes and other construction equipment located southwest quadrant of 
airport. Advise on initial contact information Kilo" 
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Scenario National B 


The aircraft will start near LAWRENCEVILLE VOR (LVL). 

LAWRENCEVILLE, victor-157 RICHMOND VOR, V-376 WASHINGTON VOR. 

ATIS X-ray is active. 


Amendment Series 1 : OK 

Reference the ATC station to MANGE (15 miles after LVL)intersection. 

When the aircraft reaches MANGE (LVL+15), do the following: 


YOU: " (callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEN D AMENDMENT #! 

YOU: "(callsign), direct WASHINGTON VOR, expect vectors to MOUNT 
VERNON VISUAL Runway 3-6." 

This amendment, is fine If he rejects, it, tell him to maintain current ame ndment 

Amendment Series 2: Routing into WX 

Reference the ATC station to RICHMOND (RIC) VOR. 

When the aircraft reaches RICHMOND, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #2 

YOU: "(callsign), direct CASANOVA VOR, direct ARMEL VOR. After 
ARMEL, expect vectors to ROSSLYN Localizer Directional Aid 
Runway 1-8. Descend and maintain one-four thousand" 

Note: ATIS Yankee now active 


IF THE PILOT DOES NOT REJECT THE AMENDMENT: 

reference the ATC station to RICHMOND VOR (KlC)When the airciaft 

is 30 miles out of RICHMOND 

YOU: "(callsign), clearance amendment, " 

when the pilot responds ready: 

ARM AND SEND AM ENDMENT #3 

YOU: "(callsign), direct COLIN intersection. After COLIN expect 
vectors to ROSSLYN Localizer Directional Aid runway 1-8. Cross 
COLIN at and maintain one-one thousand." 
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IF THE PILOT DOFS REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT 32 

YOU: "(callsign), direct COLES intersection. After COLES expect 
vectors to ROSSLYN Localizer Directional Aid runway 1-8. Cross 
COLIN at and maintain one-one thousand." 

Amendment Series 3: Illogical routing 
Reference the ATC station to TAPPA intersection. 

When the aircraft reaches five miles from TAPPA, do the following: 

YOU: "(callsign), clearance amendment, " 

when the pilot responds ready: 

ARM AND SEND AMENDMENT #4 — -™~- 
YOU: "(callsign), direct HARCUM VOR, direct NOTTINGHAM VOR. 
After NOTTINGHAM, expect vectors to ROSSLYN Localizer 
Directional Aid Runway 1-8. Cross NOTTINGHAM at and maintain 
8000 and 2-5-0 knots" 


IF THE PILOT UQES. NOT REJECT THE AMENDMENT: 

reference the ATC station to COLIN. When the aircraft is 5 miles from 

COLIN 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENT M 
YOU: "(callsign), direct NOTTINGHAM VOR. After 
NOTTINGHAM expect vectors to ROSSLYN Localizer Directional 
Aid runway 1-8. Cross NOTTINGHAM at and maintain 8000 and 2- 
5-0 knots" 

IF THE PILOT DOFS REJECT THE AMENDMENT: 

YOU: "(callsign), turn left heading 040" 

wait about 10 seconds 

ARM AND SEND AMENDMENTS 

YOU: "(callsign), direct NOTTINGHAM VOR. After 

NOTTINGHAM expect vectors to ROSSLYN Localizer Directional 
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Aid runway 1-8. Cross NOTTINGHAM at and maintain 8000 and 2- 

5-0 knots” „ 

This s cenario will terminate at NOTTINGHAM. — . 

ATIS-X: 

"Washington National airport information X-ray. 1950 Zulu. 4000 Scattered, Est 
Ceiling 10000 Broken, visibility 8 miles. Temp: 85 Dewpoint: 76. Wind 
calm. Altimeter 29.45. Visual approach RWY 36, Rwy 33 in effect. 
Departing RWY 3. Windshear advisories in effect. Advise on initial 
contact information X-ray" 

ATIS-Y 

"Washington National airport information Yankee. 2017 Zulu. Measured Ceiling 
2100 Broken, 5000 Overcast, visibility 2 miles in rain showers. Temp: 77 
Dewpoint: 74. Wind 190 at 14 gusting to 21. Altimeter 29.14. 

ROSSLYN LDA approach RWY 18 in effect. Departing RWY 15. 
Windshear advisories in effect. Advise on initial contact information 
Yankee" __ 
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Scenario Newark B 


The aircraft will start near WILLIAMSPORT VOR. The initial clearance is: 
WILLIAMSPORT, SLATE RUN 6 BWZ arrival. ATIS Foxtrot is active. 

Amendment Series 1 : Routing into WX 

Reference the ATC station to HAYED intersection. 

When the aircraft reaches HAYED, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #1 

YOU: "(callsign), direct ALLENTOWN VOR, victor 6 SOLBERG VOR, 
after SOLBERG, expect vectors to ILS runway 4 right." 

Note: ATIS Golf now active ‘ 


IF THE PILOT QQES. NOT REJECT THE AMENDMENT: 

reference the ATC station to WHITT intersection. When the aircraft is 
directly south (i.e. BRG-360) of WHITT: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #2 

YOU: "(callsign), direct STILLWATER VOR, direct SPARTA VOR. 
After SPARTA expect vectors to ILS runway 4 right. Cross SPARTA 
at and maintain one-one thousand." 

IF THE PILOT DOES REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT #2 

YOU: "(callsign), direct STILLWATER VOR, direct SPARTA VOR. 
After SPARTA expect vectors to ILS runway 4 right Cross SPARTA 
at and maintain one-one thousand." 

Amendment Series 2 : OK 

Reference the ATC station to STILLWATER VOR (STW). 

When the aircraft is 20 miles from STILLWATER, do the following: 
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YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ftPM ANJn SENfV AMENDMENT #3 

YOU: "(callsign), direct SPARTA VOR; after SPARTA expect vectors for 
ILS runway 4 right. Cross SPARTA at and maintain one-one 
thousand" 

This amendment, is OK, but_ lei him deviate as necessary 

Amendment Series 2: Illogical routing 
Reference the ATC station to SPARTA VOR (SAX). 

When the aircraft is 15 miles from SPARTA, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEN D AMENDMENT #4 

YOU: "(callsign), direct HUGUENOT VOR, direct JOELL intersection. 

After JOELL, expect vectors to ILS Runway 4 right. Cross JOELL at 
and maintain 7000. Reduce speed 2-5-0 knots" 


IF THE PILOT UQES. NHL REJECT THE AMENDMENT: 

reference the ATC station to SPARTA. When the aircraft is at SPARTA 
YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #5 

YOU: "(callsign), direct JOELL intersection. After JOELL expect 
vectors to ILS runway 4 right. Cross JOELL at and maintain 7000. 
Reduce speed 2-5-0 kn ots." 

IF THE PILOT DOES REJECT THE AMENDMENT: 

YOU: "(callsign), turn right heading 1-6-0" 

wait about 10 seconds 

ARM AND SEND AMENDMENT #5 

YOU: "(callsign), direct JOELL intersection. After JOELL expect 
vectors to ILS runway 4 right. Cross JOELL at and maintain 7000. 
Reduce speed 2-5-0 knots." 

This s cenario will tenninate_ at JOELL. 

ATIS-F: 

"Newark airport information Foxtrot. 1752 Zulu. 3500 Scattered, Measured 
Ceiling 9000 Broken, visibility 6 miles. Temp: 85 Dewpoint: 77. Wind 
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calm. Altimeter 29.38. Visual approach RWY 22L, RWY 1 1 in effect. 
Departing RWY 22R. Windshear advisories in effect. Heavy bird activity 
all quadrants of airport. Advise on initial contact information Foxtrot" 

ATIS-G 

"Newark airport information Golf. 1830 Zulu. Measured Ceiling 3000 Broken, 
9000 Overcast, visibility 3 miles in rain showers. Temp: 78 Dewpoint: 76. 
Wind 050 at 16 gusting to 27. Altimeter 29.21. ILS approach RWY 4R in 
effect. Windshear advisories in effect. Heavy bird activity all quadrants of 
airport. Advise on initial contact information Golf' 





Scenario Philadelphia B 


The aircraft will start near SHERL intersection. The initial clearance is: SHERL, V- 
139 BRIGS, Cedar Lake Arrival (VCN). ATIS Mike is active. 

Amendment Series 1 : Routing into WX 

Reference the ATC station to PLUME intersection. 

When the aircraft reaches PLUME, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #1 

YOU: "(callsign), direct SEA ISLE VOR, direct WOODSTOWN VOR, 
after WOODSTOWN expect vectors to ILS runway 2-7 right." 

Note: ATIS November now active 


IF THE PILOT DOES NOT REJECT THE AMENDMENT : 

reference the ATC station to DRIFT intersection.Wto the aircraft is at 

DRIFT 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #2 

YOU: "(callsign), direct DIXIE intersection, direct YARDLEY VOR. 
After YARDLEY expect vectors to ILS runway 2-7 right. Cross 
YARDLEY at and maintain one -zero thousand and 2-5-0 knots." 

IF THE PILOT QQES. REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SE ND AMENDMENT #2 

YOU: "(callsign), direct DIXIE intersection, direct YARDLEY VOR. 
After YARDLEY expect vectors to ILS runway 2-7 right. Cross 
YARDLEY at and maintain one-zero thousand and 2-5-0 knots." 

Amendment Series 2 : OK 

Reference the ATC station to DIXIE intersection. 

When the aircraft is 20 miles from DIXIE, do the following: 


86 



YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #3 

YOU: "(callsign), direct YARDLEY VOR, after YARDLEY expect vectors 
to ILS runway 2-7 right. Cross YARDLEY at and maintain 10000 
and 2-5-0 knots." 

This amendment^ is OK. Ifhe rejects it, tell him to maintain current^ clearance^ 

Amendment Series 3 : Illogical routing 

Reference the ATC station to ROBBINSVILLE VOR (RBV). 

When the aircraft is at RBV, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #4 

YOU: "(callsign), direct SOLBERG VOR, direct NORTH PHILADELPHIA 
VOR. After NORTH PHILADELPHIA, expect vectors to ILS 
Runway 2-7 right. Cross NORTH PHILADELPHIA at and maintain 
4000." 


IF THE PILOT QQES. NQI REJECT THE AMENDMENT: 

reference the ATC station to YARDLEY. When the aircraft is 10 miles 
from YARDLEY. 

YOU: " (callsign), clearance amendment" 

when the pilot responds ready: 

ARM AND SEND AMENDMENT 15 

YOU: "(callsign), direct NORTH PHILADELPHIA VOR. After 
NORTH PHILADELPHIA expect vectors to ILS runway 2-7 right. 
Cross NORTH PHILADELPHIA at and maintain 4000." 

IF THE PILOT OOFS REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT #5 

YOU: "(callsign), direct NORTH PHILADELPHIA VOR. After 
NORTH PHILADELPHIA expect vectors to ILS runway 2-7 right. 
Cross NORTH PHILADELPHIA at and maintain 4000." 

This s cenario wil[ tenninate cu NORTH PHILLY (PNE)._ . , ... . 

ATIS-M: 
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"Philadelphia International airport information Mike. 2052 Zulu. 4000 Scattered, 
Measured Ceiling 9500 Broken, visibility 5 miles in haze. Temp. 91 
Dewpoint: 80. Wind light and variable. Altimeter 29.58. Visual 
approach RWY 9R, RWY 9L in effect. Departing RWY 9L. DuPont 
VORTAC out of service until 0100 Zulu. Advise on initial contact 

information Mike" 

ATIS-N 

"Philadelphia International airport information November. 2127 Zulu. Measured 
Ceiling 3000 Broken, 9500 Overcast, visibility 3 miles in rain showers. 
Temp: 77 Dewpoint: 76. Wind 270 at 17 gusting to 24. Altimeter 29.31. 
ILS approach RWY 27R in effect. Departing RWY 27L. Windshear 
advisories in effect DuPont VORTAC out of service until 0100 Zulu. 
Advise on initial contact information November" 



Scenario Bradley C 

The aircraft will start at SPARTA VOR (SAX). The initial clearance is: SAX, V-2-4- 
9er WEETS INTERSECTION, V-2-0-5 BRADLEY VOR ATIS Sierra is active. 

Amendment Series 1: Wrong destination- SYRACUSE 

Reference the ATC station to SHAFF intersection. 

When the aircraft reaches SHAFF, do the following: 

YOU: "(callsign), clearance amendment, " 

when the pilot responds ready: 

ARM AND SEND AMENDMENT #1 

YOU: "(callsign), direct WEETS, victor 4-8-3 FAYET intersection, after 
FA YET expect vectors to ILS runway 2-7" 


IF THE PILOT DOES. NOT REJECT THE AMENDMENT: 

reference the ATC station to FLOSI intersection.IV/ie/j the aircraft is at 

FLOSI 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #2 
YOU: "(callsign), direct KINGSTON VOR, direct JUDDS 
intersection. After JUDDS expect vectors to ILS runway 6. Cross 
JUDDS at and maintain seven thousand and 2-5-0 knots." 

IF THE PILOT DOES. REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 
wait about 10 seconds 
ARM AND SEND AMENDMENT SI 
YOU: "(callsign), direct KINGSTON VOR, direct JUDDS 
intersection. After JUDDS expect vectors to ILS runway 6. Cross 
JUDDS at and maintain seven thousand and 2-5-0 knots." 

Amendment Series 2: OK 

Reference the ATC station to KINGSTON VOR (IGN). 

When the aircraft is 15 miles from KINGSTON, do the following: 
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YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #3 

YOU: "(callsign), direct JUDDS intersection, after JUDDS expect vectors to 
ILS Runway 6. Cross JUDDS at and maintain 7000 and 2-5-0 knots." 

This amendment is OK. If he rejects^ it, maintain current^ clearance ’. 

Amendment Series 3: Routing into WX 

Reference the ATC station to PAWLING VOR (PWL). 

When the aircraft is DIRECTLY SOUTH OF PWL, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEN D AMENDMENT #4 

YOU: "(callsign), direct SOARS intersection, after SOARS expect vectors to 
ILS Runway 6. Cross SOARS at and maintain 7000. Reduce speed 2- 
5-0 knots." 


IF THE PILOT DOES NOT REJECT THE AMENDMENT: 

reference the ATC station to SOARS. When the aircraft is at SOARS. 
ARM AND SEND AMENDMENT #5 

YOU: "(callsign), turn left heading 0-6-0, descend and maintain 

7000." 

IF THE PILOT WES REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEN D AMENDMENT #6 

YOU: "(callsign), turn left, heading 0-9-0. Descend and maintain 

7000." 

This s cenario^ will terminate about W miles from KBDL. 

ATIS-S: 

"Bradley airport information Sierra. 1750 Zulu. Measured Ceiling 2500 Broken, 
6000 Overcast, visibility 3 miles in light rain showers. Temp: 77 
Dewpoint: 75. Wind 040 at 12 gusting to 18. Altimeter 29.52. ILS 
approach RWY 6 in effect. Windshear advisories in effect. Advise on 
initial contact information Sierra" 
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Scenario National C 


The aircraft will start at ALLENTOWN VOR (ABE). The initial clearance is: ABE, 
V-12 LANCASTER VOR, V-499 BALTIMORE VOR ATIS Oscar is active 

Amendment Series 1 : OK 

Reference the ATC station to EAST TEXAS VOR. 

When the aircraft reaches EAST TEXAS, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT#! - - * 

YOU: "(callsign), direct LANCASTER VOR, direct BALTIMORE VOR, 
after BALTIMORE expect vectors to RIVER VIS UA L runway 1-8." 

This amendment^ is QIC If it is rejected, have him stay on his curreni clearance^ 

Amendment Series 2: Wrong destination- HARRISBURG INTL. 

Reference the ATC station to FLOAT intersection. 

When the aircraft reaches FLOAT, do the fottomhg: 

YOU: "(callsign), clearance amendment, " 

when the pilot responds ready: 

AR M A N D SE N D AME ND MENT 1 2 

YOU: "(callsign), direct LANCASTER VOR, direct BAARN intersection, 
after BAARN expect vectors to ILS runway 1-3." 



IF THE PILOT DOES NOT REJECT THE AMENDMENT: 

reference the ATC station to LANCASTER VOR.^/iw the aircraft is 
about 10 miles from LANCASTER; 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #3 

YOU: "(callsign), direct LANCASTER VOR, direct BALTIMORE 
VOR. After BALTIMORE expect vectors to River Visual runway 1- 
8. Cross BALTIMORE at and maintain 8000 and 2-5-0 knots." 

IF THE PILOT DOES REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 
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wait about 10 seconds 

ARM AND SEN D AMENDMENT #3 

YOU: "(callsign), direct LANCASTER VOR, direct BALTIMORE 
VOR. After BALTIMORE expect vectors to River Visual runway 1- 
8. Cross BALTIMORE at and maintain 8000 and 2-5-0 knots." 

Amendment Series 3: Routing into WX 
Reference the ATC station to TRISH intersection. 

When the aircraft is at WEST of TRISH, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT M 

YOU: "(callsign), direct DATED intersection, after DATED expect vectors 
to River Visual Runway 1-8. Descend and maintain 8000. Reduce 
speed 2-5-0 knots." 


IF THE PILOT DOES. NOT REJECT THE AMENDMENT: 

reference the ATC station to DATED. When the aircraft is 5 miles from 

DATED. 

YOU: "(callsign), clearance amendment" 
when the pilot responds ready: 

ARM AND SEND AMENDMENTS 

YOU: "(callsign), turn left, head l-9er-0. Descend and maintain 

60 00. Reduce 2-1-0 knots." 

IF THE PILOT WEE REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT M 

YOU: "(callsign), turn left, heading 2-1-0. Descend and maintain 
6000. Reduce speed 2-1-0 knots." 

This s cenario will terminate about 10 milesjrom DCA. __ 

ATIS-O: 

"Washington National airport information Oscar. 2347 Zulu. Measured Ceiling 
4000 Broken, 7000 Overcast, visibility 3 miles in light rain showers. 
Temp: 76 Dewpoint: 76. Wind l-9-0@8. Altimeter 29.65. River Visual 
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Scenario LaGuardia C 

The aircraft will start at ODESA intersection. The initial clearance is: ODESA, V-4- 
4-5 LAGUARDIA VOR. A 775 Romeo is active. 

Amendment Series 1 : OK 

Reference the ATC station to DUPONT (DQO) VOR. 

When the aircraft reaches DQO, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #1 

YOU: "(callsign), direct NANCI intersection, direct LAGUARDIA VOR, 
expect vectors to ILS runway 4." 

This amenment is OK. If rejected, mountain current, clearance 

Amendment Series 2: Wrong destination- TETERBORO 
Reference the ATC station to STEFE intersection. 

When the aircraft reaches 8 miles from STEFE, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AM ENDMENT #2 

YOU: "(callsign), direct TETERBORO VOR, after TETERBORO expect 
vectors to ILS runway 6. Descend and maintain one-one thousand. 


IF THE PILOT DOES. NQ1 REJECT THE AMENDMENT: 

reference the ATC station to SOMTO intersection. When the aircraft is 
about 5 miles from SOMTO: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENTS 

YOU: "(callsign), direct ROBBINSVILLE VOR; after 

ROBBINSVILLE, expect vectors to ILS runway 4. Cross Robbinsville 

at and maintain one-one tho usand." 

IF THE PILOT DQES. REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 


94 



wait about 10 seconds 


ARM A N D..SENP A M E ND MENT # 3 
YOU: "(callsign), direct ROBBINSVDLLE VOR; after 
ROBBINSVILLE, expect vectors to ILS runway 4. Cross Robbinsville 
at and maintain one-one thousand." 

Amendment Series 3: Rounting into WX 

Reference the ATC station to ROBBINSVILLE VOR (RBV). 

When the aircraft is 15 miles from RBV, do the following: 

YOU: "(callsign), clearance amendment, " 

when the pilot responds ready: 

ARM AND SEND AMEND M ENT M 

YOU: "(callsign), direct COLTS NECK VOR, after COLTS NECK expect 
vectors to ILS Runway 4. Descend and maintain 9000. Reduce speed 
2-5-0 knots." 


IF THE PILOT WES M2I REJECT THE AMENDMENT: 

reference the ATC station to COLTS NECK. When the aircraft is about 
5 mi from COLTS NECK. 

ARM AND SEND AMENDMENT #5 

YOU: "(callsign), turn left heading 0-2-0 descend and maintain 7000. 
Reduce speed 2-1-0 knots." 

IF THE PILOT WES REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SEND AMENDMENT #6 

YOU: "(callsign), turn left, heading 0-5-0. Descend and maintain 
7000. Reduce speed 2-1-0 knots." 

This s cenario will term ^ nate obout 10 miles from KLGA, 

ATIS-R: 

"New York LaGuardia information Romeo. 0150 Zulu. Measured Ceiling 900 
Broken, 7000 Overcast, visibility 1 and 1/2 miles in rain showers. Temp: 
68 Dewpoint: 65. Wind 050@5. Altimeter 29.54. ILS RWY 4 approach 
in effect, departing runway 31. Noise abatement procedures runway 31 in 
effect after 0200 Z. Advise on initial contact information Romeo" 



95 







Scenario Philadelphia C 

The aircraft will start at MARTINSBURG VOR (MRB). The initial clearance is: 

MRB, V-l-6-6 DUPONT VOR. ATIS Victor is active. 

Amendment Series 1: Wrong destination- BRADLEY 

Reference the ATC station to RUANE intersection. 

When the aircraft reaches RUANE, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM ANT) SEND AMENDMENT #1 

YOU: "(callsign), direct LANCASTER VOR, victor 3-9er GREKI 

intersection, af ter GREKI expect vectors to ILS runway 6« 

IF THE PILOT QQES. VAT RE JECT THE AMENDMENT: 

reference the ATC station to BINNS intersection. When the aircraft is 
about 10 miles from BINNS. 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #2 

YOU: "(callsign), direct PADRE intersection, direct MODENA 
VOR; after MODENA, expect vectors to CONVERGING ILS-2 
runway 9 right. Cross Modena at and maintain 7000 and 2*1-0 knots. 

IF THE PILOT QQES. REJECT THE AMENDMENT: 

YOU: "(callsign), standby 1" 

wait about 10 seconds 

ARM AND SE ND AMENDMENT #2 

YOU: "(callsign), direct PADRE intersection, direct MODENA 
VOR; after MODENA, expect vectors to CONVERGING ILS-2 
runway 9 right. Cross Modena at and maintain 7000 and 2-1-0 knots." 

If the pilot requests direct MODENA, say you can expect that in 5 minutes. 

Amendment Series 2: OK 

Reference the ATC station to PADRE intersection. 

When the aircraft is at 20 miles from PADRE, do the following: 
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YOU: "(callsign), clearance amendment, " 

when the pilot responds ready: 

AR M AND SE ND AME N DME N T S 

YOU: "(callsign), direct MODENA VOR, after MODENA expect vectors to 
CONVERGING ILS-2 Runway 9 right. Cross MODENA at and 
maintain 7000 and 2-1-0 knots." 

This amendment^ is QIC I£he rejects^ it, tell him to fly current^ clearance. . - 

Amendment Series 3: Rounting into WX 

Reference the ATC station to MODENA (MXE) VOR. 

When the aircraft is at 15 miles from MODENA, do the following: 

YOU: "(callsign), clearance amendment, " 
when the pilot responds ready: 

ARM AND SEND AMENDMENT #4 

YOU: "(callsign), direct DUPONT VOR, after DUPONT expect vectors to 
CONVERGING ILS-2 Runway 9 right." 



97 








Appendix B: Sample observation Sheet 

SUBJECT: 

MODE: Verbal Text Graphical 
Readback?: YES (2/3) NO (1/2) 

Amendment series 1 : OK 1 

Amendment series 2: OK 1 

Amendment series 3: OK 1 


Questions: Scale of 1 to 7: efficiency (1: unsafe-loo long 4: like current ATC 

7: optimum efficiency) 

situational awareness_ (1: unsafe-out of loop 4: like current ATC 

7: optimum awareness) 

total effectiveness (1: unsafe-info/time 4: like current ATC 

7: optimum presentation) 


RUN#: 

FMC: AUTO MANUAL 
2 3 NOT 

2 3 NOT 

2 3 NOT 


Pros: 


Cons: 
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APPENDIX C: 


Scenario Rotation information 


Key To Scenario Rotation information 

1) Run # is the sequence which the runs were accomplished 

2) Mode is the delivery mode: V = Verbal, T = Textual, G = Graphical 

3) Scenario is the scenario code (see Table 4.1) 

4) FMS is the method of FMS programming: auto = automated, man = manual 

5) Readback is whether a readback is part of the procedure 

6) Procedure # refers to Table 3.1 

7) Notes : 

No WX = weather event was not a valid test 
No RTE = routing event was not a valid test 

Neither = neither event in the scenario was a valid test OR run was incomplete 


| SUBJECT 1 , r 

RutT^^^^^MoH^^cen^o FMS Readback Procedure # Notes 


1 

V 

b.phl 

auto 

yes 

2 

2 

V 

a. bos 

auto 

no 

1 

3 

V 

c.dca 

man 

yes 

4 

4 

V 

a.bwi 

man 

no 

3 

5 

T 

b.dca 

auto 

no 

1 

6 

T 

a.jfk 

man 

yes 

4 

7 

T 

c.phl 

auto 

yes 

2 

8 

G 

b.ewr 

man 

yes 

4 

9 

G 

c.bdl 

auto 

no 

1 

10 

G 

a.lga 

auto 

yes 

2 
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SUBJECT 2 


Run # Mode Scenario FMS Readback Procedure # Notes 


1 

V 

b.bos 

man 

yes 

4 


2 

V 

c.lga 

auto 

no 

1 


3 

V 

a.bwi 

auto 

yes 

2 


4 

V 

c.bdl 

man 

no 

3 


5 

T 

b.ewr 

auto 

yes 

2 


6 

T 

a.jfk 

auto 

no 

1 


7 

T 

c.dca 

man 

yes 

4 


8 

G 

b.phl 

auto 

no 

1 


9 

G 

a.jfk 

man 

yes 

4 

Neither 

10 

G 

c.phl 

auto 

yes 

2 


ISUBJECT 3 





Run# 

Mode Scenario FMS 

Readback 

Procedure # 

Notes 

1 

V 

b.dca 

auto 

no 

1 


2 

V 

a.jfk 

man 

yes 

4 


3 

V 

c.phl 

auto 

yes 

2 


4 

V 

b.phl 

man 

no 

3 


5 

T 

c.dca 

man 

yes 

4 



T 

a.lga 

auto 

no 

1 


7 

T 

b.bos 

auto 

yes 

2 


8 

G 

a.bos 

auto 

yes 

2 


9 

G 

c.lga 

auto 

no 

1 


10 

G 

b.ewr 

man 

yes 

4 


ISUBJECT 4 





Run # 

Mode Scenario FMS 

Readback 

Procedure # 

Notes 


1 

T 

b.ewr 

auto 

no 

i 


2 

T 

a.lga 

man 

yes 

4 

No RTE 

3 

T 

c.bdl 

auto 

yes 

2 


4 

G 

b.bos 

man 

yes 

4 


5 

G 

c.lga 

auto 

no 

i 


6 

G 

a.bos 

auto 

yes 

2 


7 

V 

b.dca 

auto 

yes 

2 


8 

V 

a.bwi 

auto 

no 

1 

No WX 

9 

V 

c.phl 

man 

yes 

4 

Neither 

10 

V 

a.jfk 

man 

no 

3 

Neither 
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iSUBJECT 5 

Run# Mode Scenario FMS Readback Procedure# Notes 


1 

T 

b.phl 

auto 

no 

i 


2 

T 

a.bos 

man 

yes 

4 

NoRTE 

3 

T 

c.dca 

auto 

yes 

2 


4 

G 

b.dca 

auto 

no 

1 


5 

G 

a.bwi 

man 

yes 

4 

NoRTE 

6 

G 

c.phl 

auto 

yes 

2 


7 

V 

b.ewr 

man 

yes 

4 


8 

V 

c.bdl 

auto 

no 

i 


9 

V 

a.jfk 

auto 

yes 

2 


10 

V 

c.lga 

man 

no 

3 

Neither 

SUBJECT 6 





Run # 

Mode 

Scenario FMS 

Readback 

Procedure # 

Notes 

1 

T 

c.lga 

man 

yes 

4 


2 

T 

a.bwi 

auto 

no 

1 


3 

T 

b.bos 

auto 

yes 

2 


4 

G 

a.jfk 

auto 

no 

1 


5 

G 

c.bdl 

man 

yes 

4 


6 

G 

b.ewr 

auto 

yes 

2 


7 

V 

b.phl 

auto 

no 

1 

NoRTE 

8 

V 

a.lga 

man 

yes 

4 


9 

V 

c.dca 

auto 

yes 

2 


10 

V 

b.dca 

man 

no 

3 



ISUBJECT 7 1 

Run# Mode Scenario FMS Readback Procedure# Notes 


1 

G 

b.dca 

auto 

yes 

2 

2 

G 

c.phl 

man 

yes 

4 

3 

G 

a.jfk 

auto 

no 

1 

4 

V 

b.phl 

auto 

yes 

2 

5 

V 

a.lga 

auto 

no 

1 

6 

V 

c.dca 

man 

yes 

4 

7 

V 

a.bos 

man 

no 

3 

8 

T 

b.bos 

auto 

yes 

2 

9 

T 

a.bwi 

auto 

no 

1 

10 

T 

c.lga 

man 

yes 

4 
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SUBJECT 8 

Run # Mode Scenario FMS Readback Procedure # Notes 


1 — 

i 

cT 

b.ewr 

auto 

no 

i 



2 

G 

a.lga 

man 

yes 

4 


5 1 

t :zZ 
E. -5 

3 

G 

c.bdl 

auto 

yes 

2 



4 

V 

b.bos 

man 

yes 

4 



5 

V 

c.lga 

auto 

no 

i 



6 

V 

a. bos 

auto 

yes 

2 


* ; 

7 

V 

c.phl 

man 

no 

3 



8 

T 

b.dca 

auto 

no 

1 


- 

9 

T 

a.bwi 

man 

yes 

4 



10 

T 

c.dca 

auto 

yes 

2 

NoRTE 

1:1 

SUBJECT 9 






Run# 

Mode 

Scenario FMS 

Readback 

Procedure # 

Notes 


1 

G 

a.bos 

auto 

no 

1 



2 

G 

c.dca 

man 

yes 

4 



3 

G 

b.phl 

auto 

yes 

2 



4 

V 

b.dca 

auto 

no 

1 



5 

V 

a.bwi 

man 

yes 

4 



6 

V 

c.phl 

auto 

yes 

2 



7 

V 

b.ewr 

man 

no 

3 



8 

T 

c.bdl 

auto 

no 

1 


" 

9 

T 

a.jfk 

auto 

yes 

2 



10 

T 

b.bos 

man 

yes 

4 

Neither 
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appendix D: Statistical analysis 


The statistical method used for the performance measurements was a Non- 
parametric Paired test [9]. This method is appropriate for a “within subjects” 
experimental design. This appendix will explain the general method and will give a 
numerical example. 

Between treatments of data (i.e. the categories being compared), there were two 
tests which were performed: 1) the data was either detected or not detected, and 2) the 
data was either detected initially or not detected initially. (In the numerical example, the 
treatments are the graphical and verbal mode of delivery, and the test was the initial 
detection performance for routing events.) 

The data was first divided into the appropriate treatments, and the performance for 
each subject was recorded for the treatments. Care was taken to ensure that the 
comparisons between treatments was fair: if a particular data point was unusable for one 
treatment, the corresponding data point for the other treatment was removed from the 
analysis. 

After the scores for each subject were recorded, ties between treatments for 
individual subjects were discarded, yielding an adjusted number of subjects. The 
performance of the remaining subjects was compared as to which treatment gave a higher 
score, and this number was tested against the values in Table D.l. If the number of 
subjects whose scores for one treatment exceeded the tabulated value, a statistical 
difference at the corresponding tabulated level was assigned to that test. 



TABLE D.l: TEST VALUES FOR THE NON-PARAMETRIC PAIRED TEST [9] 

Adjusted # Of Subjects 

# of Subjects Needed to Yield 
95% Significance For a 
Treatment 

# of Subjects Needed to Yield 
99% Significance For a 
Treatment 

6 

5 

- 

7 

6 

- 

8 

7 

7 

9 

7 

8 


For the numerical example (comparing the initial detection performance for 
routing events between the graphical and verbal modes), the individual subject scores for 
each treatment (i.e. graphical and verbal) are shown in Table D.2. Since Subjects 4 and 9 
had the same performance for both treatments, their scores are ignored. Of the seven 
remaining subjects, six garnered a higher score for graphical, rather than verbal, delivery. 
Since this number (i.e. 6) equals the tabulated value for the remaining subjects, the 
performance is judged to be significant at the 95% level. 


TABLE D.2: DATA FOR NUMERICAL EXAMPLE 



Subject # 

1 

2 

3 

4 

5 

6 

IB 

8 

9 

Graphical: Initial 
Detections - Routing 

3 

1 

1 

0 

1 

1 

i 

2 

2 

Verbal: Initial 
Detections - Routing 

2 

0 

0 

0 

0 

0 

0 

3 

2 


104 


































REFERENCES 


[1] Knox, Charles E., and Charles H. Scanlon. “Flight Tests Using Data Link For Air 

Traffic Control and Weather Information Exchange”. SAE Aerospace Technology 
Conference and Exposition SAE 901888. Long Beach, CA. October 1990. 

[2] Waller, Marvin C., and Lohr, Gary W. A Piloted Simulation Study of Data Link ATC 

Message Exchange. NASA Technical Paper 2859. Hampton, VA. 1989. 

[3] Wickens, Christopher D. Engineering Psychology and Human Performance. 

Harper-Collins. Champaign-Urbana, IL (no information on publication date 
provided). 

[4] Chandra, Divya. “An Evaluation of Automation for Flight Path Management in 

Transport Category Aircraft”. MIT Man-Vehicle Laboratory. Cambridge, MA. 
August 1989. 

[5] Wanke, Craig, Divya Chandra, R. John Hansman, and Steven R. Bussolari. “A 

Comparison of Voice and Datalink For ATC Amendments and Hazardous Wind 
Shear Alerts”. 4th International Symposium on Aviation and Space Safety. 
Toulouse, France. November 1990. 

[6] Kems, Karol. Datalink Communication Between Controllers and Pilots: A Review 

and Synthesis of the Simulation Literature. MITRE Corporation MP-90W00027. 
McLean, VA. July 1990. 

[7] Midkiff, Alan H., et al. Informal personal interviews conducted between June 1990 

and October 1991. 

[8] Kuchar, James K., and R. John Hansman. “An Exploratory Study of Plan View 

Terrain Displays for Air Carrier Operations”. Preprint, International Journal of 
Aviation Psychology. 

[9] Siegel, Andrew F., Practical Business Statistics. Irwin. Boston, MA. 1990. 

[10] Bateman, Don. Flight Into Terrain and the Ground Proximity Warning System: 

Including 150-Plus Accidents and Events. Sundstrand Data Control. Engineering 
Report 070-4251. February, 1991. 


105 



